Learning From Data

Learning From Data by Yaser S. Abu-Mostafa et al is good intro to both a theoretical and practical approach to understanding modeling. Let’s make things clear, this is a textbook – not a passive read. At about 200 pages, it on the slim side for a textbook, but as the authors note in the preface, the book is “a short course, not a hurried course”.Complexity:Despite not having much modeling experience, the book was relatively easy for me to understand, although some of it did go over my head. It is a very good choice as an introduction to the field. The authors do an incredible job of weaving narrative into the knowledge early on, although this becomes less common later in the book. By that I mean most sections contain examples relating the topic to real life applications which prevents the math from becoming too abstract. And there is a good amount of math. Before reading this book, I would recommend having taken multivariable calculus since they use gradients and other things fairly frequently. If you have taken enough math but think you are a bit rusty the book takes care of that. The authors have been kind enough to include a “Table of Notation” at the back of the book to let you refresh yourself if you come across an unfamiliar symbol. So if you forget what a downward pointing triangle means, there is still hope for you!Chapter 1: The Learning Problem.In this chapter the authors provide the basic background to learning from data. As stated earlier, this is where they do some of their best work connecting the theory to real life examples. The writing style in some of these examples is almost like prose which makes it a much more enjoyable and memorable read. They summarize some of the main types of learning and define some of the key terms and principles like error and noise.Chapter 2: Training versus Testing.This chapter begins to explain what the theory of generalization is, the associated error, and numerical approximations of generalizations. I would still categorize this chapter as background knowledge that will be used more in later parts of the book.Chapter 3: The Linear Model.This is really the meat of the book. If you want to quickly learn how to do a regression then jump straight to this chapter. It covers both linear and logistic regression and also touches on nonlinear transformations.Chapter 4: Overfitting.This chapter deals with the more advanced aspects of modeling. As the authors put it “the ability to deal with overfitting is what separates professionals from amateurs”. While I can safely say that I am still an amateur, it was nice to be exposed to some of the more advanced concerns of the field. Overfitting, for those who don’t know, is trying to fit to the data more than is needed. This is often done by using more degrees of freedom than necessary to make a model i.e. making a 10th order approximation of data whose original function is actually only 2nd order. To someone new to modeling it can be very tempting to increase the order of an approximation because it might seem that higher order = higher accuracy. This section of the book does a good job of explaining how that isn’t the case by introducing the idea of an overfit penalty, the increase in error from overfitting a curve.Chapter 5: Three Learning Principles.This chapter is different. Very different. Instead of continuing to introduce other types of models, the authors decide to use the fundamentals taught in earlier chapters to talk about three key principles that are useful in modeling: Occam’s razor, sampling bias, and data snooping. Occam’s razor is something that is well-known. The book uses Albert Einstein’s explanation of the term, “An explanation of the data should be made as simple as possible, but no simpler.” This relates to the previous chapter’s problem of overfitting. Sampling bias talks about errors in modeling that come from having data that is not representative of the overall population. Data snooping talks about deciding to make a prediction after looking at data rather than before. This section was probably the most interesting to read. Unlike the previous few chapters the authors return to relying on real world examples. They even use the historical example of the false prediction of the 1948 US presidential election between Truman and Dewey. Again this makes the information in this chapter much more memorable and was by far my favorite chapter of the book.Problems/Exercises:I only briefly skimmed over a few of the exercises and problems included in this book but they did help improve my understanding of the topics. According to the authors the provided practice problems are “easy” so if you want more of a challenge you will need to look elsewhere.Criticisms:In the middle chapters of the book, the authors use fewer real world examples and prose style writing. This is extremely unfortunate because, to someone unfamiliar with the field, it provided a hook to draw you into the more math heavy sections. Also, while the authors are very detailed and thorough in explaining different theories and types of models, they do not do a great job of listing the strengths and weaknesses of each. If I was required to make one of the models explained in this book, I probably could, but if I was asked to choose which model would be best for a given situation I would probably be unable to do so. They should have been clearer about what models are used in different situations and provided guidelines for selecting which model to use (beyond Occam’s razor). This would better connect the material to real world use and be more beneficial to the readers.Conclusion:Learning From Data provided a quick but thorough overview of modeling and machine learning. If you would like to learn more about the subject and have the required math background, it is a very good place to start. It will give you the background, main models, errors, and principles necessary for you to not only learn the language of the field but also critique and even create your own models. I highly recommend it.Score: 4.5/5

I've owned a much loved copy of this book for several years. It does an incredibly good job of explaining why, and when, statistical learning methods work. It also introduces the mathematical prerequisites required to go out and further explore the field. Having said that, I found the topic coverage somewhat limited, and the approach surprisingly abstract. For example, there is no mention of software methods, or of popular learning models like neural networks. The text strongly hints that readers who solve the included problems will gain additional and valuable insight, but solutions to these are not readily available. This makes self study just that much harder. It's stuff like this that left me disinclined to make a strong recommendation for this book in the past.Recently I discovered an open secret that many other people seem to have known for the better part of a decade. A website associated with the book provides additional online chapters. Also included is a freely available video lecture series from one of the authors. It was recorded live in a real classroom at Caltech in 2012. The accompanying homework and final exams, with solution sets, are also available to all!The new chapters available online seemed to be written in same accessible style of the book. These chapters definitely address my complaint regarding limited topic coverage. The homework problems also seem to put a kibosh on the criticism that the book is too abstract. They seem particularly well chosen, and if you actually solve them you will definitely gain real experience with these learning techniques, all of it in the language and software platform of your choice. The solution keys tell you when you need to go back and work on a problem some more, or if you got the solution right. The video lectures I have watched are all spectacularly good. It's rare to find such a gifted teacher, and I even found myself watching him cover familiar ground just for the pleasure of observing the exposition.So, is it acceptable to give this book a "five stars!" rating for things that are not actually in it? I don't really know, and frankly, I think the question is irrelevant here. If you are willing to use the online resources, this book is amongst the very best introductions to machine learning available. If you choose to ignore the online resources, it is still an outstanding introduction to the mathematical foundations of the subject. While I wish there was a second edition with the new chapters, I can settle for my own printed copies of the downloaded documents for now!

The book accompanies Caltech's online course of the same name https://telecourse.caltech.edu/index.phpPretty much in line with the online lectures. The course covers machine learning in general and focuses on the theory of learning with introductory material on various learning algorithms incorporated into the chapters as they become relevant. I'm giving this 4 stars because it is indeed a short course on learning from data as the cover says. This is not an in-depth book on learning algorithms although it does of course cover some of these in reasonable depth but always from a computer science angle towards the theory of learning rather than in a practical applied 'engineering' light. The book and the online course itself is actually of very high quality (despite being free); the professor's lectures are very well structured, organised and explained. Finally, the price is OK but a little high given its total knowledge content - although addmitedly I am maybe a little miserly :)

I can't pretend to have spent nearly as much time with this book as I've signed up to. It arrived this afternoon, and it got at least 20 minutes of my time looking ahead at what Professor Abu-Mostafa is going to say to the world tomorrow. But from page 15 to 27, here is what I can observe.A minor point: I love the color examples within the book. When I was in school, we didn't get textbooks like this.The tone of the text is very sympathetic in the direction of one who is interested in the subject. That is, if you need to learn from data, the text considers what you will need to know to succeed. The reader's need to understand is prioritized much higher than the reader's need to "be educated".Although I've been out of college for two decades, I have no problem following what the text has to say, or the direction it's headed in. I don't abstract things magnificently, my calculus isn't that hot, and I slid through school without a probability class as such. I love math, but I'm not a powerhouse at it compared to the peers who once sat next to me. But I can follow this text, understand the exercises, and I conjecture that I can work the problems which are meant to be worked.For a non-academic, I have considerable on-the-ground machine learning background. I've done a lot of backpropagation network training in my day in character recognition, acoustics, finance, and similar disciplines. This is where I find the text valuable: it can build on what I already know, and it starts in a place I'm already familiar with. So if this is your field or one of your fields, the book will make not just a good textbook but a useful reference.So let's talk about what this book is not, and perhaps AMLbook will publish something else for us. Don't expect a course in neural networks here. They're not in the table of contents ("hmmm..."), but you'll find them in the index pointing you to the epilogue. Support vector machines get the same treatment. Now support vector machines are a little new for me; they came into their own after I finished college. Neural networks, however, got an earlier start.What I can say is that this text is intentionally tool-neutral. I just read 13 pages that weren't about neural networks per se, but they covered vital groundwork that I learned via hands-on experience with real neural networks. My grasp of some of the "why" components of what I know is already improved.I am not in love with whatever chemical processes went into the manufacture of this book. Perhaps it came off the press so recently that I'm noticing that, and the effects will fade. For right now, it's a definite don't-cuddle-with volume which drives me to the bathroom sink when I put it down. I can live with this; it's a small price to pay for the knowledge.The amount I paid for this text was astonishingly low. No doubt the authors have thought this through carefully, but this text will hold its own against others that cost three to six times what I paid.I apologize if this is not the best-informed review. I'm hitting send a little early, because at the present time, you don't have many alternatives to read.

If you are a beginner in ML, buy this book! If you have a quantitative background and are planning to dabble in ML, buy the book. I fall in the latter category, but am a ML novice; and recently, started to learn through ML books and online resources. Until recently, I couldn't quite build a mental framework within which to view and place ML approaches, justifications, performance, etc., nor could I quite contrast ML against "classical" data analysis approaches (oh well, why can't I just fit the data, and carefully deal with interpolation/extrapolation matters, etc. etc.). As I explored the subject through other texts & online material, largely, it seemed a choppy sea of techniques flotsam, a confusing landscape of differing terminology, and, I couldn't quite "put my finger" on the subject, nor could I conceive a framework to view the ML approach. And then, fortuitously, I came across this book! I believe it provides a much needed principled approach to the subject, introducing intuitive terminology, suggesting a line of thought through sequencing of material, and systematically covering the core essentials; all this, in a manner that nicely balances readability & mathematical rigor. After a quick read, I do have a mental framework to place ML developments (and no doubt, it will get tweaked & refined over time). Since I'm a novice, I risk being shot down for making such bold statements, however, I suspect I'm probably not far from the truth in saying that there is a need for such an expository text. I wouldn't be surprised if there aren't a ton of other ML beginners who are as appreciative of this book, as I am. Thank you for this wonderful book!Incidentally, the authors also provide a free online course covering the material in the text, and a few additional topics covering the basics of neural networks, SVMs, kernel methods, and the use of RBFs. The book, along with these additional online video lectures, are a near perfect mix to get the beginner going!

This book is a phenomenal resource. It might very well be in a league of its own. In my opinion there are a handful of machine learning books that I've come across that really stand out from the rest: Introduction to Statistical Learning (Hastie, Tibshirani), Elements of Statistical Learning (Hastie, Tibshiani), Applied Predictive Modeling (Kuhn, Johnson), Python Machine Learning (Raschka), and then this book. I've also heard great things about Machine Learning: A Probabilisitc Perspective (Murphy) and Pattern Recognition and Machine Learning (Bishop), though I haven't yet read those texts. ISL, APM, and PML each are terrific with a blend of theoretical but mostly geared towards practical applications. ESL is more focused on theory and a deep understanding of the math behind the algorithms/functions (i.e. how do they work, which is essential for understanding when to apply which algorithm). This text is different in that it's more focused on the problem of learning: is learning possible? how do we learn? and how do we learn well? This is an essential foundation that everyone in the field needs to have.The text is impressive in and of itself; however, what really makes this book stand out is that the authors have set up a website dedicated to the text with a blog, dynamic e-chapters (covering NN, SVM, Similarity Methods, Linear Algebra, and various techniques to aid in learning (i.e. dimension reduction)). Its much like a virtual classroom where you may ask questions regarding the book or anything ML in general and other reviewers and the text authors are there to answer your questions. There are very high quality videos (lectures) along with slides for each chapter. Thus, when you buy the text you gain access to each of these resources (lectures are free on YouTube whether you purchase the book or not). ISL also has its own course online with lecture videos (which can be found on YouTube). I would say that anyone who is starting out in the field needs these two texts: LFD and ISL. From each you get your theory and practical foundations along with lectures and learning resources. The back of the book also contains a very comprehensive dictionary of notations used throughout the book.

Reading this book gives you the feeling that you are in a classroom and Prof. Abu-Mostafa is teaching. Very well written, very clearly explained facts and methods, and the authors know exactly how to make the heavy-duty stuffs interesting and clearly explainable in class-room setting. I enjoyed reading every page of this book. Not to forget that this book is also accompanied by 'the best set of video lectures' on machine learning taught by Prof. Abu-Mostafe at Caltech, available online, with a wide range of discussions on the topics, problems and solutions.One may think that this book is a bit thin in terms of its contents. To me it is important to know why I am doing certain things rather than to know what are the possible things I can do. This book is along the line of why, where and of course how we use machine learning, and provides tools whenever necessary, without sacrificing mathematical rigor. It is meant to be a short course as the title says. Even if you have read other books on machine learning, reading this book is worthwhile; at the least, it will revitalize your interest in the subject and give you a clear(er) perspective. Therefore, I feel that this book is a must-read for everyone. Happy reading!

The book helped me quite a lot when I was enrolled in Caltech's online course on machine learning taught by Prof. Yaser last summer (2012). It is more heavy in concepts than in practical examples, but shows clearly the "boundaries" and "gotchas" you have when doing general ML.Although scarce in numeric examples, it has many good advice spread around. The ML techniques which are very well explained in the book are those related with linear or linearized (through nonlinear transformations) models (linear regression and classification, perceptron, use of gradient in minimization,...), although in the Caltech's course Neural Network and SVM models and techniques have been very well presented.The models taught in the book serve as test beds for treating the perennial problems: when is learning possible? How to overcome data noise, sampling bias, overfitting and data snooping issues in practical situations? These fundamental concerns are transverse to all ML models and techniques. These are the matters where this book shines.The book is high-quality (nice paper and color pictures), and is sold at a very low price: it is a must-have for ML wizards :-) (and wannabees).

Shipping took awhile, but overall I am happy. You also need access to read the remaining 3 chapters online as they are not present in the physical copy.

A must-have for every machine learning student.

I like the book very much. As a said, is very good for beginners

It takes a dedicated teacher to write a book like this. No wonder the author is so passionate on screen which is also reflected in the book. We would love a second edition which incorporates all the extra chapters which is now available only as an online edition (that is ehybibratebit as 4/5 otherwise this surely deserves a 5/5)

Bought the book second-hand from the US. It assumes a lot of prior knowledge of Maths and programming. Despite high production quality, I wouldn't recommend it at all to machine learning beginners unless you are very comfortable with engineering mathematics (functions, statistics, calculus) and at least one programming language. I have already done Prof Andrew Ng's Coursera ML course, and still couldn't follow its first chapter after the first few pages. This book is intended to be the classroom text for the courses the authors take in their universities, so maybe it is not supposed to be used for self-education. It might become more comprehensible with teachers explaining things in parallel. Might be good for reference after you already know ML.