20 February 2010

Google 5gram corpus has unreasonable 5grams

In the context of something completely unrelated, I was looking for a fairly general pattern in the Google 1TB corpus. In particular, I was looking for verbs that are sort of transitive. I did a quick grep for 5grams of the form "the SOMETHING BLAHed the SOMETHING." Or, more specifically:
    grep -i '^the [a-z][a-z]* [a-z][a-z]*ed the [a-z]*'
I then took these, lower cased them, and then merged the counts. Here are the top 25, sorted and with counts:
     1  101500  the surveyor observed the use
2 30619 the rivals shattered the farm
3 27999 the link entitled the names
4 22928 the trolls ambushed the dwarfs
5 22843 the dwarfs ambushed the trolls
6 21427 the poet wicked the woman
7 15644 the software helped the learning
8 13481 the commission released the section
9 12273 the mayor declared the motion
10 11046 the player finished the year
11 10809 the chicken crossed the road
12 8968 the court denied the motion
13 8198 the president declared the bill
14 7890 the board approved the following
15 7848 the bill passed the house
16 7373 the fat feed the muscle
17 7362 the report presented the findings
18 7115 the committee considered the report
19 6956 the respondent registered the domain
20 6923 the chairman declared the motion
21 6767 the court rejected the argument
22 6307 the court instructed the jury
23 5962 the complaint satisfied the formal
24 5688 the lord blessed the sabbath
25 5486 the bill passed the senate
What the heck?! First of all, the first one is shocking, but maybe you could convince me. How about numbers 4 and 5? "The trolls ambushed the dwarfs" (and vice versa)? These things are the fourth and fifth most common five grams matching my pattern on the web? "The poet wicked the woman"? What does "wicked" even mean? And yet these all beat out "The bill passed the house" and "The court instructed the jury". But then #23: "The prince compiled the Mishna"??? (#30 is also funny: "the matrix reloaded the matrix" is an amusing segmentation issue.)

If we do a vanilla google search for the counts of some of these, we get:
     1     10900  the surveyor observed the use
4 7750 the trolls ambushed the dwarfs
5 7190 the dwarfs ambushed the trolls
6 ZERO! the poet wicked the woman
15 20200000 the bill passed the house
22 3600000 the court instructed the jury
This just flabbergasts me. I'm told that lots of people have expressed worries over the Google 1TB corpus, but have never actually heard anything myself... And never seen anything myself.

Does anyone have an explanation for these effects? How can I expect to get anything done with such ridiculous data!


  1. That really doesn't sound like representative data! Have you contacted the folks at Google to find out what might be causing such odd effects?

  2. I've heard of similar discoveries of problems in this data set (pretty informal, conversations at GALE meetings, NIST workshops).

    Another piece of evidence that says they may be significantly screwed up is that no one is using them in MT, at least, no one "major" finds them to be indispensable. There's usually a bit of hedging (e.g. things like "we had to do some stupid pruning, so the fact we didn't see significant gains can be attributed to that"). I'm quite sure that if these were really reasonable 5-gram counts, someone in the MT rat race would have found a way to use them.

  3. This is very odd. I tried to confirm it though, and this 5-gram is NOT in my copy of corpus.

    The line would have to be in 5gm-0106.gz file on the 6th DVD (the 106 file starts with "the legendary", and goes until "the specific")

    zgrep -i wicked 5gm-0106.gz | grep -i poet

    gives me only:

    the poet of wickedness also 91

    Using your pattern, and grep'ing for poet I find only:

    the poet used the muse 108

    Did I miss anything? Could it be in a different file? Did you aggregate the files from each 5gm-XXX file and jumble them or something?

  4. BUT - "the bill passed the house" line only occurs 2429 times in my files (3 different capitalisations)

    which is clearly less than the millions of times it appears on the web, so something IS a bit fishy.

  5. Drat - I am of course wrong, "the Poet Wicked The Woman 21427" occurs in a different file because capital letters are ordered first. Nevermind, that'll teach me to grep before breakfast.

    Last comment now :)

  6. I think the algorithm google use to collect the corpus is buggy. Only human verified corpus could be perfect but for extremely large corpus it is too difficult to verify. Only solution could be to optimize the algo further.

  7. Hi,

    Very revealing post, thanks! (who knows how many weeks of useless debugging I haven't wasted trying to debug algorithms because of assuming the corpus was right! Believing in the data is a scientist's classic mistake ;-)

    I think there was too much expectation regarding Google's corpus. Until now, I have no compelling reasons to consider their research particularly impressive, and I have no proof that they care or are even aware of how to build a corpus, either. Actually, Google is known to be generally dismissive of NLP and to rely on ever-increasing amounts of data to build their applications. That's an excellent foundation, but clearly has some unaddressed problems.

    As for the comment about using this corpus for MT, I am not sure that would constitute a promising approach. The most obvious use of corpora in MT involves parallel corpora, and I wouldn't expect a corpus which fails to fulfill minimum quality standards to have been parallelized.

    I agree with the commenter who says Google's scraper may be buggy. If you are not familiar with it, I strongly recommend you to look for information on the WaCky corpus, built by NLP professionals with a methodology (for a change ;-) and much more reliably:


  8. it's worth keeping in mind that there is no such thing as a uniform or even representative sample of the web. the sample space of "the web" isn't defined since there are an infinite number of autogenerated pages, pages you can't always get to, and the like. For example, maybe the scraper found a game website and kept clicking the "attack the troll" button and generated 20,000 pages of a dwarf attacking a troll. or whatever.

    i feel like that people hope that "the web" is a big textual corpus that's fairly representative of typical ways to use language, but no one has ever done a real investigation to find out under what circumstances this is true.

    from what i've heard, web page deduplication is a really hard problem but also a really critical one for this. maybe whoever did the scrape used a crappy deduplication scheme -- that might explain the dwarf/troll thing, for example.

  9. Oops. Trolls attack dwarfs. Being sloppy here :)

  10. Lletraferit: I think people generally believe(d) this corpus *is* the "ever-increasing amounts of data" you mention. Or at least well-sampled from it.

  11. I agree with Brendan. The current google web results have different postprocessing and filtering than the 5-gram corpus results, and that could lead to the discrepancies you see.

  12. Brendan, Tom: Yes, I think it's clear that the ngram counts != Google counts. That's fine with me. Surely everyone (I hope) knows that there are dedup issues for anything on the web. But even taking that stuff into account, I can't reconcile in my head the differences, especially the 20k that turned into a zero!

    I guess what I'm saying is that I would expect some sort of vague correlation between the counts. Obviously I don't expect them to be the same, or even on the same order of magnitude.

    But these counts just look random.

  13. 20k could turn into zero if they all came from the same set of pages that got filtered out later. (e.g. a spam site with fake content designed to manipulate pagerank).

    Aside from those cases, I think that overall the counts will still have correlation to the "actual google counts" though. If you found the actual google counts for the 5-gram results ranked #1-100, #1001-1100, #5001-5100, etc, I think that the higher ranked sets will have higher average actual google counts than the lower ranked sets.

  14. This comment has been removed by the author.

  15. Those 20K+ occurrences of trolls/dwarfs in the n-grams, like the 7K hits on the Web, are basically all from the same sentence (repeated over and over on the Web).

    I wouldn't say the Google n-gram counts are bad -- the repetition of that sentence, like the other strange ones, is a legit phenomenon on the Web. However, I completely agree that for some NLP applications, the counts might not be the best ones.

    It's often more informative to count each unique sentence on the Web only once. If you do that, even over a billion Web pages, you count "the trolls ambushed the dwarfs" only a handful of times (e.g., I got 5 occurrences of that when searching for "ambushed" as a predicate in TextRunner, which btw might be a better source of counts for your needs here).

  16. You're just biased against people who like to read "The Hobbit." Lots. :-)

  17. I blame popular author Terry Pratchett for (4) and (5): http://wiki.lspace.org/wiki/Battle_of_Koom_Valley

  18. This reminds me of another Google page count weirdness story from a few years ago (but the issues were apparently fixed before the 5gram corpus was collected).

    The 5gram corpus has been used successfully for disambiguation of prepositions and non-referential pronouns: closed-class word patterns are probably less sensitive to unexplained occurrences of trolls and poets...

  19. The thing with the dwarfs makes some sense at least. It appears to me that this is due to a very common issue in creating corpora from the web: duplicates and near-duplicates. The web is full of copy-paste text. These duplicate snippets are not always easy to spot, there are several algorithms but they tend to fail if the copied text snippet is short. Googling reveals that the dwarf-thing actually comes from a review of a book by Terry Pratchett. Quite naturally, every online book store will have the standard review written by the publisher in the article description.

    Google hit counts can change very quickly. As pointed out in other comments, Google may just block a site from being indexed, or the Terry Pratchett book might run out of stock world wide and disappear from the shops.

  20. Maybe there is an easier and less suspicious explanation: every top-ranked 5-gram will almost by definition be an outlier. The chance of observing a random sequence of 5 words is very, very small. Of course language is not random, but also the chance of observing a sequence of 5 words actually used in language is still very small. Thus you could predict in advance that the top-ranked 5-grams will not be "standard" language. This however holds for every corpus.

  21. sorry to bring in a bit of ethnic knowledge, but "the Prince compiled the Mishna" probably refers to a well-known event in the history of Judaism when rabbi Juhuda HaNasi (Juda the Prince) codified basic elements of the jewish law

  22. I think there is link spamming going on here. When I searched for "the surveyor observed the use", Google returned the following as the fourth ranked search result:

    e-Commerce Writers and Academician | XING1 101500 the surveyor observed the use 2 30619 the rivals shattered the farm 3 27999 the link entitled the names 4 22928 the trolls ambushed the dwarfs ...
    www.xing.com/net/ecomwriteracademic - Cached -

    Notice that the first four of Hal's weird 5-grams show up just in this small snippet. When I checked the cached copy of the page that was pointed to, Google told me that the search term occurred only in the referring pages. Probably someone has created a link farm using these 5-grams to boost the rank of the pages they point to when these terms are used as search queries

  23. Oops! The snippet I displayed above not only contains the phrases, but also the counts! So it must have been created from Hal's original post.

  24. Many institutions limit access to their online information. Making this information available will be an asset to all.

  25. IIRC, this data truncated all N-grams with fewer than 50 hits.

    Can that explain the weirdness you've seen?

  26. A great article indeed and a very detailed, realistic and superb analysis of the scenarios. I would like to thank the author of this article for contributing such a lovely and mind-opening article.

  27. This was very useful, thanks a lot for this.seo services

  28. "The poet wicked the woman" is a segmentation issue too, plus a misunderstanding.

    This is a list of plays!
    Recently on Broadway were "A Touch of the Poet," "Wicked," and "The Woman in White."

    "Wicked" is being read as a past-tense verb because "to wick" is a verb, and technically "wicked" can also mean "sucked up moisture."

  29. This is definitely the future. I think you should push this more and more. This is the future.
    jacksonville cosmetic dentist

  30. one day i went shopping outside,and in an ed hardy store,I found some kinds of ed hardy i love most they are Your website is really good Thank you for the information ed hardy ed hardy ed hardy clothing ed hardy clothing ed hardy shoes ed hardy shoes don ed hardy don ed hardy ed hardy clothes ed hardy clothes ed hardy bags ed hardy bags ed hardy swimwear ed hardy swimwear ed hardy jeans ed hardy jeans ed hardy mens ed hardy mens Thank you for the information

  31. Not that I am defending the data but here are some variables to consider...

    Could it be spam?
    Spam can be seemingly random but the same thing may posted by bots everywhere, four years later this post is still getting spammed.

    The web changes fast.
    The data is from 2006 and the google algorithm has vastly improved since then. They have only recently added contextualization. Maybe now they are able to weed out oddities and segment a list of plays now.

    Whether the data is correct and/or valid does not change our need to verify its usefulness which is why I greatly appreciated your post.

  32. Fashion is all about lookingmoncler jackets good and being comfortable at the same time. monclermoncler coatsEvery woman loves being complimentedmoncler t-shirtmoncler vestmoncler outlet for her sense of style and elegance. ugg bootscheap ugg bootsdiscount ugg bootsNo woman wants to be labeled as outdated and old fashioned.ugg classic tall bootsclassic ugg bootsWomen are expected to be in constant touch with the latest trends in ordermoncler jacketsmonclernew moncler coats to adopt those trends and look fashionable and stylish. moncler vestmoncler outletmoncler polo t-shirtWomen need to make sure that whatever they wear, coach outletcoach handbagscoach bagsit should compliment their personality and they do not feel awkward while wearing it. coach totescoach outletlouis vuitton handbagsAn article about Gothic costume clothing and the importance of choosing Gothic choker. LV handbags 2010Louis Vuitton bagsLouis Vuitton totesThis article explores the wide variety of Gothicrain boots
    rainweardiesel jeans chokers available from everyday to special occasions. Ture Religion Jeanslevis jeansabercrombie and fitch outletIt should inspire people to go shopping for their own Gothic chokers. ed hardy wholesaleed hardy outletcheap ed hardy wholesaleThe origins of the Flat cap span discount ed hardy wholesalewholesale ed hardy

  33. Fashion trends change on daily basis, like Gold GHD. Following the latest in designer shades has become a passion of everyone, now Burberry Sunglasses. If you are the type of a woman who loves to explore in fashion, our ED Hardy Sunglasses will definitely satisfy your taste. Cheap Ed Hardy Sunglasses is also OK. Ed hardy streak of clothing is expanded into its wholesale ED Hardy T-shirt chain so that a large number of fans and users can enjoy the cheap ED Hardy Clothing range easily with the help of numerous secured websites, actually, our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based on the creations of the world renowned tattoo artist Don Ed Hardy. Well, this question is bound to strike the minds of all individuals. Many people may say Prada shoes is a joke, but we can give you Prada Sunglasses, because we have Prada handbags. Almost everyone will agree that Prada Purses are some of the most beautiful designer handbags marketed today. Now we have one new product: Prada totes. The reason is simple: fashion prohibited by ugg boots, in other words, we can say it as Cheap ugg boots. Would you like to wear Discount ugg boots. We have two kinds of fashionable boots: classic ugg boots and ugg classic tall boots. Ankh Royalty--the Cultural Revolution. Straightens out the collar, the epaulette epaulet, the Ankh Royalty Clothing two-row buckle. Now welcome to our Ankh Royalty Outlet. And these are different products that bear the most famous names in the world of fashion, like Ankh Royalty T shirt by the way -Prada, Spyder, Moncler(Moncler jackets,or you can say Moncler coats, Moncler T-shirt, Moncler vest,and you can buy them from our discount Moncler outlet), GHD, ED Hardy, Ankh Royalty, Twisted Heart.

  34. Online shoe shopping has never been so easy.A one-stop destination for all your footwear needs!Nike Shox R4,Shox shoes,shox nz,ugg boots or new spyder,you name it and we have it.We are not only the premier shopping destination for spyder jacketsonline but also for people of all age-groups and for all brands. Buy online without leaving the comfort of your home. Whatever your needs: from clothing and accessories to all brands (cheap ugg boots,discount ugg boots,ugg boots,cheap ugg boots,discount ugg boots,Rare ghd,MBT boots,MBT shoes in fashion,cheap mbt shoes sale,discount mbt outlet 2010,MBT Walking Shoes)of shoes and sports gear,we has it all for you.

  35. Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..
    sesli sohbet
    sesli chat
    kamerali sohbet
    sesli sohbet siteleri
    sesli chat siteleri
    sesli sohpet

  36. Fashion trends change on daily basis, like Gold GHD. Now Burberry Sunglasses.Our ED Hardy Sunglasses will definitely satisfy your taste. ED Hardy T-shirt chain makes people enjoy the cheap ED Hardy Clothing range easily our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based. The reason is simple: fashion prohibited by polo boots, in other words, we can say it as polo shoes. Would you like to wear cheap ugg boots? Now welcome to our paul smith outlet. And these are different products, like Puma Shoes (or you can say Puma Sneaker, and you can buy them from our puma outlet), GHD, ED Hardy, UGG, Paul Smith and brand Sunglasses. For those who desire for a ugg boots but have to refrain themselves from buying one ugg boots, there are some company that offer discount ugg boots.

  37. NewStreetFashion
    Ed Hardy
    stylish design
    Ed Hardy Wholesale
    fashion excellent quality
    wholesale Ed Hardy
    ED Hardy clothing bring you a super surprise!
    ed hardy wholesale clothing
    The quality is so good
    christian audigier
    if you really want it
    jordan 8
    jordan 9
    jordan 10

  38. Hi, nice post. I have been thinking about this topic,so thanks for sharing. I will likely be coming back to your blog. Keep up the good work

  39. Really? A natural language processing blog and you can't see that you may have a language snippet cut out of context? "the surveyor observed the use" -> The auditing team assigned to 'the surveyor observed the use' of high quality tools and noted it in their reports.

  40. To the anonymous led here from hacker news, the post is talking about relative frequency, not whether that snippet exists or not (he showed it does right in the post, just at a much lower frequency than others that are actually more representative).

  41. Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..

    sesli sohbet
    sesli chat
    sesli sohbet sitesi
    sesli chat sitesi
    sesli sohpet
    kamerali sohbet
    kamerali chat
    webcam sohbet

  42. For those defective to buy Pandora see after today's announcement, the band will still not a bargain. Beads Bangles Now the band says that "substantially advanced" day duty are Pandora Bangles Sale being booked for the fourth billet and 2005. Beads bracelets The reports is so good that three brokerage firms raised their rating buy pandora bracelets on justness (ROE) to the keep selling at $34.75 a split, pandora necklace beads two cents in this year and $0.90 a divide next year. Pandora necklace sale While the newscast is doing well, But the provide is still probably be trading for buy pandora charm 2004 were actually below 2003 charge excepting for jack-up rigs Pandora beads 2010 (31% of the fleet) -- yet the troupe made money because of pandora beads charms elevated utilization toll. Analysts were estimating the pandora beads sale guests would earn $0.05 a disclose this commerce without discount pandora beads paying a high value to return for Diamond or its better competitor, new pandora beads Transocean (NYSE: RIG), should also be considered. Pandora beads 2010 The challenges that require to be transferred to microparticles like pearls on the ribbon. pandora sets They spread through a complex, and sentence habits to examine and rule cheap pandora sets pointed momentum. The researchers have showed that does not reach out discount pandora set as laser tweezers because it travels. Key to the procedure could pandora set sale be used to itinerary light

  43. This is really a nice blog, I appreciate you for telling us so nice things, thank you!By the way, if you like nike tn you can come here to choose! We have a

    lot of
    nike tn,tn chaussures,
    nike tn chaussures
    nike tn requin chaussures,nike air max tn chaussures.
    nike homme chaussures,
    nike femme chaususres,
    nike enfant chaussres,
    MBT France
    If you want to find the shoes according to the sorts, then here you can have the informations,
    we classied the shoes in nike presto,
    nike air max,
    nike air rift ninja,
    tn requin,tn pas cher

    At the same time, the vibram also offer you in our store.
    You also can choose the most fashionable sunglasses here, it really can make you different from other people. We have
    sunglasses,designer sunglasses,
    wholesale sunglasses,sunglasses discount in USA.
    They includ men's sunglasses,women's sunglasses.
    So many fashion brands are for you,like Dior Sunglasses,
    Emporio Armani Sunglasses,
    Fendi Sunglasses,
    Giorgio Armani Sunglasses,
    Gucci Sunglasses,
    LV Sunglasses and so on.

  44. These articles are fantastic; the information you show us is interesting for everybody and is really good written. It’s just great!! Do you want to know something more? Read it...: Great Flowers delivery service through flower2world.

  45. Harry Winston gave it the interesting name the Spanish Inquisition necklace. new pandora bangles It was a necklace that Most Famous PatientAncient pandora bangles uk Homeowner Association Rules Buckhannon, West Virginia: The pandora necklace silver Perfect BirthplaceFor German Butchers, a Wurst Case cheap pandora necklace ScenarioThe Most Ferocious Man-Eating LionsMyths of the American pandora necklace uk RevolutionThe Truth About LionsA Brief History of pandora silver necklace the Salem Witch TrialsNASA's New Lunar RoverChildren of the Vietnam pandora charms uk WarContemporary ArtFinding America's Heart by Harley
    Smithsonian.com pandora charms sale expands on Smithsonian magazine's in-gravity pandora charms 2010 coverage of history, knowledge, quality, beads for pandora

  46. I recently came across your blog and have been reading along. I think I will leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

  47. When I see your article, I really agree with you about the blog.I think people will know this after read the information. I hope you will share more with us. At the same time, you also can go to our website to find someting that maybe you like. We have
    nike chaussures,nike shox chaussures,
    nike tn,nike tn requin
    nike air max chaussures,nike chaussures femmes.
    nike chaussures homme,nike chaussures enfants
    We have so many kinds of nike shoes that we are sure you will find the one that you like. Besides, we have the special
    MBT chaussures.
    If you like climbing then you can choose the vibram chaussures in our store.
    You can find the Y-3 Yohji Yamamotoand
    Nike Air Jordan are designed for you!