26 January 2010

A machine learner's apology

Andrew Gelman recently announced an upcoming talk by John Lafferty. This reminded me of a post I've been meaning to write for ages (years, really) but haven't gotten around to. Well, now I'm getting around to it.

A colleague from Utah (not in ML) went on a trip and spent some time talking to a computational statistician, who will remain anonymous. But let's call this person Alice. The two were talking about various topics and at one point machine learning came up. Alice commented:

"Machine learning is just non-rigorous computational statistics."
Or something to that effect.

A first reaction is to get defensive: no it's not! But Alice has a point. Some subset of machine learning, in particular the side more Bayesian, tends to overlap quite a bit with compstats, so much so that in some cases they're probably not really that differentiable. (That is to say, there's a subset of ML that's very very similar to a subset of compstats... you could probably fairly easily find some antipoles that are amazingly different.)

And there's a clear intended interpretation to the comment: it's not that we're not rigorous, it's that we're not rigorous in the way that computational statisticians are. To that end, let me offer a glib retort:
Computational statistics is just machine learning where you don't care about the computation.
In much the same way that I think Alice's claim is true, I think this claim is also true. The part of machine learning that's really strong on the CS side, cares a lot about computation: how long, how much space, how many samples, etc., will it take to learn something. This is something that I've rarely seen in compstats, where the big questions really have to do with things like: is this distribution well defined, can I sample from it, etc., now let me run Metropolis-Hastings. (Okay, I'm still being glib.)

I saw a discussion on a theory blog recently that STOC/FOCS is about "THEORY of algorithms" while SODA is about "theory of ALGORITHMS" or something like that. (Given the capitalization, perhaps it was Bill Gasarch :)?) Likewise, I think it's fair to say that classic ML is "MACHINE learning" or "COMPUTATIONAL statistics" and classic compstats is "machine LEARNING" or "computational STATISTICS." We're really working on very similar problems, but the things that we value tend to be different.

Due to that, I've always found it odd that there's not more interaction between compstats and ML. Sure, there's some... but not very much. Maybe it's cultural, maybe it's institutional (conferences versus journals), maybe we really know everything we need to know about the other side and talking wouldn't really get us anywhere. But if it's just a sense of "I don't like you because you're treading on my field," then that's not productive (either direction), even if it is an initial gut reaction.


Bob Carpenter said...

I think the issue here is largely semantic.

Was the inventor of L-BFGS a computational statistician, a machine learnologist, or merely a numerical computationalist? What about all the heavy algorithmic lifting involved in sampling (e.g. Casella and Roberts' book)? What about the folks who implemented BUGS? Or something like lmer or even the basic regression in R (or SAS or Stata)? Which field gets to claim the bootstrap or Metropolis-Hastings or Gibbs sampling?

Are all those convergence theorems in NIPS papers more computational stats than machine learning, given that they're not about algorithms per se?

PS. I really think one of the big cultural differences is one Lafferty mentioned, saying "the goal of machine learning is to develop computer programs that predict well, according to some measure of risk or accuracy". That seems to make it a branch of decision theory focusing on algorithms.

Statisticians seem more focused on the analysis of fixed data sets rather than generalization behavior to unseen data. They tend to care more about causality and significance for this reason, because they're trying, in some sense, to use statistics to draw scientific conclusions.

Fernando Pereira said...

To put Bob's and John's points in summary form: machine learning cares about cost and risk, statistics cares about truth (hypothesis testing). In "The Emergence of Probability," Ian Hacking wrote about the conceptual struggle to separate probability (truth) from expectation (risk/reward). My tongue-in-cheek cynical conclusion is that probability allowed statisticians to save their necks in political perilous times by leaving the prediction and prescription business to economists and other risk-takers.

hal said...

I totally (mostly) agree with Bob and Fernando -- in fact, that's what I tell students on day one of my machine learning class!

But I don't think it's the whole story. There are whole parts of machine learning that don't care about predictions (a large subset of unsupervised learning, for instance). And similarly there are parts of statistics that care about predictions (as Bob points out, some parts of decision theory).

So I think it's a good first approximation, but it's not the whole story.

Anonymous said...

Although it seems to be unfashionable within some of the more statistically oriented parts of the community today, machine learning used to worry about some larger goals of 'intelligence' - something statisticians never really worried about or had much to say about...

Lawyer Nevada said...

Your article was quite intriguing and the information quite useful. Will check your site often to see other great posts you make! Regards

Nevada Attorneys, Nevada
, Nevada Law Firms,
Nevada Law Offices, Nevada
Legal Services
, Attorneys
In Nevada
, Nevada Lawyer
, Nevada Attorney
, Nevada Accident Attorneys, Nevada Administrative & Governmental Law Attorneys, Nevada Adoption Attorneys, Nevada Agricultural Law Attorneys, Nevada Appeals Attorneys, Nevada Arbitration & Mediation Services, Nevada Arbitration & Mediation Services Attorneys, Nevada Asbestos Diseases Attorneys, Nevada Asset Protection Attorneys, Nevada Attorneys, Nevada Attorneys&#; Information & Referral Services, Nevada Attorneys&#; Support Services, Nevada Banking & Investment Law Attorneys, Nevada Bankruptcy Attorneys, Nevada Business Services, Nevada Child Abuse Law Attorneys

Attorney New Mexico said...

As a Newbie, I am always searching online for articles that can help me. Thank you

State of New Mexico Lawyer Directory,
New Mexico Attorney Search,
New Mexico Lawyers Search, Find
A New Mexico Attorney Lawyers
, New Mexico Civil Law Attorneys, New Mexico Collection Law Attorneys, New Mexico Computers & Technology Law Attorneys, New Mexico Constitutional Law Attorneys, New Mexico Construction Law Attorneys, New Mexico Consumer Protection Attorneys, New Mexico Domestic Partnerships Attorneys, New Mexico Drug Charges Attorneys, New Mexico DUI/DWI Attorneys, New Mexico Education Law Attorneys, New Mexico Elder Law Attorneys, New Mexico Land Use & Zoning Attorneys, New Mexico Landlord & Tenant Law Attorneys, New Mexico Legal Information Services, New Mexico Legal Services, New Mexico Trial Attorneys, New Mexico Vehicular Accident Attorneys, New Mexico Whistleblower Attorneys, New Mexico Workers&#; Compensation Attorneys, New Mexico Wrongful Death Attorneys, New Mexico Wrongful Termination Attorneys

Unknown said...

Interesting post! We've been trying to foster just such a collaboration here at UCL in the UK at our "Centre for Computational Statistics and Machine Learning":


Unknown said...

nike air max 90
nike air max 95
nike shoes
nike air
nike air shoes
nike air max tn
nike air rift
nike shox r4
sports shoes
nike air rifts
nike air rift trainer
nike air max 360
nike shox nz
puma cat
air max trainers
mens nike air max
nike air
puma mens shoes
puma shoes
puma speed
nike shoes air max
nike shoes shox
air shoes
Lucyliu IS Lucyliu
nike shoe cart
puma future
levis jeans
nike rift shoes
cheap nike air rifts
bape shoes
cheap puma
nike rift
jeans shop
diesel jeans

Unknown said...

cheap nike shox
cheap sport shoes
nike tn dollar
ed hardy ugg boots
ed hardy love kills slowly
ed hardy clothing us
ed hardy clothing
cheap ed hardy
cheap ed hardy clothing
ed hardy clothes
ed hardy wholesale
ed hardy clothing
ed hardy t shirts
ed hardy shirts
ed hardy uk
ed hardy t shirts
ed hardy shirts
ed hardy hoodies
cheap nike max ,。
puma future cat
ed hardy ugg boots.
ed hardy love kills slowly boots.
ed hardy love kills slowly.
ed hardy polo shirts.
cheap ed hardy clothing,.
ed hardy shirts .
ed hardy t shirts.,.,.

Unknown said...

Fashion trends change on daily basis, like Gold GHD. Following the latest in designer shades has become a passion of everyone, now Burberry Sunglasses. If you are the type of a woman who loves to explore in fashion, our ED Hardy Sunglasses will definitely satisfy your taste. Cheap Ed Hardy Sunglasses is also OK. Ed hardy streak of clothing is expanded into its wholesale ED Hardy T-shirt chain so that a large number of fans and users can enjoy the cheap ED Hardy Clothing range easily with the help of numerous secured websites, actually, our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based on the creations of the world renowned tattoo artist Don Ed Hardy. Well, this question is bound to strike the minds of all individuals. Many people may say Prada shoes is a joke, but we can give you Prada Sunglasses, because we have Prada handbags. Almost everyone will agree that Prada Purses are some of the most beautiful designer handbags marketed today. Now we have one new product: Prada totes. The reason is simple: fashion prohibited by ugg boots, in other words, we can say it as Cheap ugg boots. Would you like to wear Discount ugg boots. We have two kinds of fashionable boots: classic ugg boots and ugg classic tall boots. Ankh Royalty--the Cultural Revolution. Straightens out the collar, the epaulette epaulet, the Ankh Royalty Clothing two-row buckle. Now welcome to our Ankh Royalty Outlet. And these are different products that bear the most famous names in the world of fashion, like Ankh Royalty T shirt by the way -Prada, Spyder, Moncler(Moncler jackets,or you can say Moncler coats, Moncler T-shirt, Moncler vest,and you can buy them from our discount Moncler outlet), GHD, ED Hardy, Ankh Royalty, Twisted Heart.

Unknown said...

moncler jacketsmonclermoncler coatsmoncler t-shirtmoncler vestmoncler outletugg bootscheap ugg bootsdiscount ugg bootsugg classic tall bootsclassic ugg bootsmoncler jacketsmoncler
new moncler coatsmoncler vestmoncler outletmoncler polo t-shirtcoach outletcoach handbagscoach bagscoach totescoach outletlouis vuitton handbagsLV handbags 2010Louis Vuitton bags
Louis Vuitton totesrain bootsrainweardiesel jeansTure Religion Jeanslevis jeansabercrombie and fitch outlet
ed hardy wholesaleed hardy outletcheap ed hardy wholesalediscount ed hardy wholesalewholesale ed hardyGHDGHD straightenersCheap GHDDiscount GHDGHD OutletGHD New Pink

Unknown said...

Online shoe shopping has never been so easy.A one-stop destination for all your footwear needs!Nike Shox R4,Shox shoes,shox nz,ugg boots or new spyder,you name it and we have it.We are not only the premier shopping destination for spyder jacketsonline but also for people of all age-groups and for all brands. Buy online without leaving the comfort of your home. Whatever your needs: from clothing and accessories to all brands (cheap ugg boots,discount ugg boots,ugg boots,cheap ugg boots,discount ugg boots,Rare ghd,MBT boots,MBT shoes in fashion,cheap mbt shoes sale,discount mbt outlet 2010,MBT Walking Shoes)of shoes and sports gear,we has it all for you.

Unknown said...

Everyone has unique requirements in terms of brands,personal specifications and choices.Whether you want to clothing online,new moncler,moncler jackets,moncler coats,moncler outlet,moncler vest,moncler polot-shirt,spyder outlet,spyder polot-shirt,spyder vest,spyder coats,cheap ed hardy wholesale,ed hardy wholesale,discount ed hardy wholesale,wholesale ed hardy,ed hardy outletyou can just take your pick, put it in the shopping cart and make your payment.FromSexy Lingerie Store,Intimate Apparel,Sexy Halloween Costumes,Sexy Underwear,cheap vibram 5 fingers, discount vibram five fingers,Vibram running shoes,vibram five fingers shoes,vibram five finger outlet ,you get them all here. The real benefit is in the discount prices.

Unknown said...

Good writing. Keep up the good work. I just added your RSS feed my Google News Reader..
New Jersey Attorneys Legal Services
, Attorney Directory New Jersey,
State Of New Jersey Lawyers,
Lawyers New Jersey, Attorneys
Of New Jersey

New Jersey Attorney Lawyer, New Jersey Lemon Law Attorneys, New Jersey Malpractice & Negligence Attorneys, New Jersey Maritime & Admiralty Law Attorneys, New Jersey Media & Communications Law Attorneys

Unknown said...

Well Whattadya know, yet another great site to add to my reader! Google blog search has you pretty well indexed it seems! you have some brilliant contents!
New Jersey Medical Malpractice Attorneys, New Jersey Military & Veterans Law Attorneys, New Jersey Native Persons Law Attorneys, New Jersey Non-Attorney Court Agents, New Jersey Nonprofit Organizations Attorneys, New Jersey Patent & Trademark Attorneys, New Jersey Personal Injury Attorneys,
New Jersey Product Liability Law Attorneys, New Jersey Property Law Attorneys, New Jersey Real Estate Attorneys

Anonymous said...

Hey very nice blog!! Man .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds also…
Alabama motorcycle insurance, Alaska motorcycle insurance, Arizona motorcycle insurance, Arkansas motorcycle insurance, Bail Bonds Referral Services motorcycle insurance, California motorcycle insurance, Colorado motorcycle insurance, Connecticut motorcycle insurance, Delaware motorcycle insurance, District Of Colombia motorcycle insurance, Florida motorcycle insurance,

Anonymous said...

This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?
This is very interesting information. I am doing some research for a class in school. and i liked the post. do you know where I can find other information regarding this? I am finding other information on this but nothing that I can use really in my paper for my final. do you have any suggestions?

Ann said...

Fashion trends change on daily basis, like Gold GHD. Now Burberry Sunglasses.Our ED Hardy Sunglasses will definitely satisfy your taste. ED Hardy T-shirt chain makes people enjoy the cheap ED Hardy Clothing range easily our ED Hardy Outlet. As we all know, in fact discount ED Hardy, is based. The reason is simple: fashion prohibited by polo boots, in other words, we can say it as polo shoes. Would you like to wear cheap ugg boots? Now welcome to our paul smith outlet. And these are different products, like Puma Shoes (or you can say Puma Sneaker, and you can buy them from our puma outlet), GHD, ED Hardy, UGG, Paul Smith and brand Sunglasses. For those who desire for a ugg boots but have to refrain themselves from buying one ugg boots, there are some company that offer discount ugg boots.

Unknown said...

Ed Hardy
stylish design
Ed Hardy Wholesale
fashion excellent quality
wholesale Ed Hardy
ED Hardy clothing bring you a super surprise!
ed hardy wholesale clothing
The quality is so good
christian audigier
if you really want it
jordan 8
jordan 9
jordan 10

Unknown said...

coats & jackets
If one knows where to, the world will give way.
north face jackets
World-class brands
moncler jackets
The top fabric produced in France is 100%
Italy's top outdoor brands
Good quality and function, more accord with fashionable design
spyder jackets
High-grade, innovation, the trend of the brand
Columbia Sportswear
The quality is so good
quiksilver jackets
Young and creative style
ski jacket

You can have a look at it.
jordan shoes
newest jordans
jordan 7
air yeezy
jordan true flight
If you like,you can contact us.
jordan 3
jordan 4
We offer different styles.
jordan 1
jordan 2
jordan 5
jordan ajf shoes
There are cheap shoes to choose
nike footwear
jordan flight 45
Good quality with low price.
air jordan 2010
Air Jordan 2009
Enjoy it!
nike trainers

Anonymous said...

I can't believe it.
abercrombie outlet
You can have a look at it.
abercrombie fitch outlet
The quality is so good
nike shoes wholesale
wholesale jordan shoes
Young and creative style
wholesale clothing
wholesale nike shoes
wholesale jordans
Young and creative style
china wholesale
There are cheap shoes to choose
jordan 6

jane simi's blog said...

Thanks very much for your suggestion.I can get a lot of information from you article.And there is also so much nice jackets for all of you,i hope you like them.
moncler jacken
moncler jackets
moncler men
moncler coats
moncler women
Thanks for you attention.

Unknown said...

ed hardy
ed hardy clothing
discount ed hardy
ed hardy discount
cheap ed hardy
wholesale ed hardy
buy ed hardy
don ed hardy


Ed Hardy Men Accessories
ed hardy man wristbands
ed hardy men bags
ed hardy men wallets
ed hardy men belts
ed hardy men caps
ed hardy men jewelry
ed hardy men socks
ed hardy men sunglasses
ed hardy men ties


ed hardy men watches
ed hardy scarves
ed hardy gloves
Ed Hardy Men Apparel
ed hardy men board shorts
ed hardy men demin
ed hardy men hoodies
ed hardy men outerwear
ed hardy men polos
ed hardy men sweater
ed hardy men knits
ed hardy men t-shirts
ed hardy underwear loungewear

Unknown said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it to a few friends of mine that I know would enjoy reading..

sesli sohbet
sesli chat
sesli sohbet sitesi
sesli chat sitesi
sesli sohpet
kamerali sohbet
kamerali chat
webcam sohbet

jacket said...

You can have a look at it.
coats & jackets
jordan shoes
The quality is so good.
abercrombie and fitch
abercrombie & fitch
Abercrombie and fitch outle

combattery84 said...

Laptop Battery
acer Laptop Battery
apple Laptop Battery
asus Laptop Battery
compaq Laptop Battery
Dell Laptop Battery
fujitsu Laptop Battery
gateway Laptop Battery
hp Laptop Battery
ibm Laptop Battery
sony Laptop Battery
toshiba Laptop Battery
APPLE M8403 battery
APPLE A1078 Battery
APPLE A1079 battery
APPLE A1175 battery 1
APPLE a1185 battery
APPLE A1189 battery
Acer aspire 5920 battery
Acer btp-arj1 battery
Acer LC.BTP01.013 battery
Acer ASPIRE 1300 battery
Acer ASPIRE 1310 battery
Acer Aspire 1410 battery
Acer ASPIRE 1680 battery
ACER BTP-63D1 battery
ACER BTP-43D1 battery
Acer lc.btp05.001 battery
Acer aspire 3000 battery
Acer Travelmate 4000 battery
ACER aspire 5560 battery
ACER BATBL50L6 battery
ACER TravelMate 240 Battery
ACER BT.00803.004 Battery

combattery84 said...

ACER Travelmate 4002lmi battery
Acer travelmate 800 battery
Acer aspire 3613wlmi battery
Travelmate 2414wlmi battery
Acer batcl50l battery
Acer Travelmate 2300 battery
ACER aspire 3610 battery
ACER travelmate 4600 battery
Dell Latitude D800 battery
Dell Inspiron 600m battery
Dell Inspiron 8100 Battery
Dell Y9943 battery
Dell Inspiron 1521 battery
Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery
Dell G5260 battery
Dell Inspiron 9200 battery
Dell Latitude C500 battery
Dell HD438 Battery
Dell GK479 battery
Dell PC764 battery
Dell KD476 Battery
Dell Inspiron 1150 battery

ylinling001 said...

I like your article, really interesting! My point is also very good, I hope you'll like:chi flat iron are a very popular choice of hair straightener.New Balance,new Blance shoes,new Blance Outlet are some of the most comfortable and stylish shoes on the market today. The designer has a whole range of shoes for all types of athletes. five finger shoes,vibram five fingers,Five fingers shoes give women the feeling of walking barefoot while still keeping the feet protected.

Anonymous said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
sesli sohbet
sesli chat
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
kameralı sohbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
sesli sex

trustme said...

This is really a nice blog, I appreciate you for telling us so nice things, thank you!By the way, if you like nike tn you can come here to choose! We have a lot of
nike tn,tn chaussures,
nike tn chaussures
nike tn requin chaussures,nike air max tn chaussures.
nike homme chaussures,
nike femme chaususres,
nike enfant chaussres,
MBT France
If you want to find the shoes according to the sorts, then here you can have the informations,
we classied the shoes in nike presto,
nike air max,
nike air rift ninja,
tn requin,tn pas cher
vibram fivefingers,
At the same time, the vibram also offer you in our store.
You also can choose the most fashionable sunglasses here, it really can make you different from other people. We have
sunglasses,designer sunglasses,
wholesale sunglasses,sunglasses discount in USA.
They includ men's sunglasses,women's sunglasses.
So many fashion brands are for you,like Dior Sunglasses,
Emporio Armani Sunglasses,
Fendi Sunglasses,
Giorgio Armani Sunglasses,
Gucci Sunglasses,
LV Sunglasses and so on.

Unknown said...

Really trustworthy blog. Please keep updating with great posts like this one. I have booked marked your site and am about to email it

to a few friends of mine that I know would enjoy reading..
sesli sohbet
sesli chat
sesli site
görünlütü sohbet
görüntülü chat
kameralı sohbet
kameralı chat
sesli sohbet siteleri
sesli chat siteleri
sesli muhabbet siteleri
görüntülü sohbet siteleri
görüntülü chat siteleri
görüntülü muhabbet siteleri
kameralı sohbet siteleri
kameralı chat siteleri
kameralı muhabbet siteleri
canlı sohbet
sesli muhabbet
görüntülü muhabbet
kameralı muhabbet
sesli sex