Abu-Mostafa Y.S., Magdon-Ismail M., Lin H.-T. Learning From Data: A short course

Pasadena: AMLbook.com, 2012. — 201 p. — ISBN: 978-1-60049-006-4.This book is designed for a short course on machine learning. It i s a short course, not a hurried course. From over a decade of teaching this material, we have distilled what we believe to be the core topics that every student of the subject should know. We chose the title 'learning from data' that faithfully describes what the subject is about , and made it a point to cover the topics in a story-like fashion. Our hope is that the reader can learn all the fundamentals of the subject by reading the book cover to cover.
Learning from data has distinct theoretical and practical tracks. If you read two books that focus on one track or the other, you may feel that you are reading about two different subjects altogether. In this book, we balance the theoretical and the practical, the mathematical and the heuristic. Our criterion for inclusion is relevance. Theory that establishes the conceptual framework for learning is included, and so are heuristics that impact the performance of real learning systems. Strengths and weaknesses of the different parts are spelled out. Our philosophy is to say it like it is: what we know, what we don't know, and what we partially know.The Learning Problem
Problem Setup
Types of Learning
Is Learning Feasible?
Error and Noise
Training versus Testing
Theory of Generalization
Interpreting the Generalization Bound
Approximation-Generalization TradeoffThe Linear Model
Linear Classification
Linear Regression
Logistic Regression
Nonlinear TransformationOverfitting
When Does Overfitting Occur?
Kegularization
ValidationThree Learning Principles
Occam's Razor
Sampling Bias
Data SnoopingEpilogue
Further Heading
Appendix. Proof of the VC Bound
Relating Generalization Error to In-Sample Deviations
Bounding Worst Case Deviation Using the Growth Function
Bounding the Deviation between In-Sample Errors