# Statistical Learning Theory, Spring Semester 2016

## Instructor

Prof. Dr. J.M. Buhmann## Assistants

Viktor WegmayrAlexey Gronskiy (contact TA)

## General Information

The ETHZ Course Catalogue information can be found here.

The course covers advanced methods of statistical learning. The fundamentals of Machine Learning as presented in the course "Introduction to Machine Learning" are expanded and, in particular, the following topics are discussed:

*Statistical Learning Theory:*How can we measure the quality of a classifier? Can we give any guarantees for the prediction error?*Variational Methods and Optimization:*We consider optimization approaches for problems where the optimizer is a probability distribution. Concepts we will discuss in this context include:*Maximum Entropy**Information Bottleneck**Deterministic Annealing*

*Clustering:*The problem of sorting data into groups without using training samples. This requires a definition of ``similarity'' between data points and adequate optimization procedures;*Model Selection:*We have already discussed how to fit a model to a data set in ML I, which usually involved adjusting model parameters for a given type of model. Model selection refers to the question of how complex the chosen model should be. As we already know, simple and complex models both have advantages and drawbacks alike;*Reinforcement Learning:*The problem of learning through interaction with an environment which changes. To achieve optimal behavior, we have to base decisions not only on the current state of the environment, but also on how we expect it to develop in the future;:

## Time and Place

Type | Time | Place |
---|---|---|

Lectures | Mon 14-16 | HG G 5 |

Exercises | Mon 16-17 | HG G 5 |

## Student Forum

The link (member approval with google e-mail would be needed). Please feel free to use it for any questions, comments to the TA's, for sharing ideas and discussing assignments, projects and anything related to SLT with other students.## News

Date | Reason |
---|---|

June 19 | Typos corrected in the tutorial on Mean-Field approximation (Variational Inference). |

May 9 | Project report template (which is in fact a NIPS paper one) is available. |

May 9 | Here's the link (member approval with google e-mail would be needed) to the Student Forum SLT. Please feel free and creative to use it for sharing ideas on home assignments, projects, etc (it's by no means a competition). |

Apr 28 | PROJECTS:
Dear all, we've made the project descriptions available in the respective section
below. Please find there the general project information
(workflow, deadlines, grading etc), as well as and project descriptions. |

Apr 20 | We started putting solutions online. Please keep track of them. |

Apr 19 | Preliminary project explanation document is updated! Here (updated) you can find the updated preliminary SLT project gradings. |

Apr 18 |
1. Here (updated) you can find preliminary
SLT project gradings. The projects themselves are expected to be delivered on
Monday, May 2nd. Or a bit earlier, time premitting. 2. We decided to put an "Sechselauten exercise" so that you could practice. It is an "joker" one in the sense that we will count it towards Testat if you hand it in but if you don't it will not count towards not getting Testat. |

Apr 11 | We started putting non-handwritten tutorials on the web. |

Mar 29 | Happy Easter to those who celebrate! We decided to put an "Easter exercise" so that you could practice. It is an "joker" one in the sense that we will count it towards Testat if you hand it in but if you don't it will not count towards not getting Testat. |

Mar 15 | Corrected some typos in Exercise 3. |

Mar 10 | Added the coed for tutorial and corrected some typos in Exercise 2. |

Mar 02 | We aplogize for a delay with the exercise, it will be delivered on 3d of March. For those needing the Testat we will provide grading relaxations due to this delay ;-) |

Feb 21 | Lecture for Feb 22 added. |

Feb 19 | Dear all, we decided to start the Tutorials from the first week, i.e. Feb 22. |

Feb 18 | Time and place added |

Jan 11 | Creation of the website. Welcome! |

## Material

Date | Lecture/Tutorial Slides | Exercise Series, Hometasks |
---|---|---|

Feb 22 | Lecture [pdf] | |

Feb 29 | Lecture [pdf] | Exercise [pdf] |

Mar 7 | Lecture [pdf] MCMC Tutorial Code [zip] |
Exercise [pdf] Solution [pdf] |

Mar 14 | Tutorial Notes [pdf] | Exercise [pdf] Solution [pdf] |

Mar 21 | Lecture [pdf] Tutorial Notes [pdf] |
Exercise [pdf] Solution [pdf] |

Mar 28 (Easter) | No class | "Extra/Joker" Exercise [pdf, handwritten] (solutions to Viktor; typed version later, sorry for the inconvenience) |

Apr 4 | Lecture [pdf] Tutorial Notes [pdf] |
Exercise [pdf] Solution [pdf] |

Apr 11 | Lecture [pdf] Tutorial Notes [pdf] |
Exercise [pdf] Solution [pdf] |

Apr 18 (Sechselauten) | No class Prelimiary Project Grading (upd) [pdf] |
"Extra/Joker" Exercise [pdf, handwritten] (solutions to Viktor; typed version later, sorry for the inconvenience) |

Apr 25 | Lecture [pdf] Tutorial Notes [pdf, updated] |
Exercise [pdf] |

May 2 | ||

May 9 | Lecture [pdf] | Exercise [pdf] Solution [pdf] |

May 16 | No class | |

May 23 | Lecture [pdf] | No exercise |

May 30 | Lecture [pdf] | Project presentations |

## Projects

Diffusion MRI bundle segmentation | Semantic clustering of nouns |
---|---|

Project description [pdf] | Project description [pdf] |

## Reading

- Prelimiary course
script (ver. 04 Aug 2015) (draft TeX by Sergio Solorzano). This script
**has not been fully checked**and thus comes without any guarantees, however, is good for getting oriented in the material. - Tikochinsky, Tishby, Levine, "Alternative Approach to Maximum-entropy Inference", Phys. Rev. A (1984).
- Duda, Hart, Stork: Pattern Classification, Wiley Interscience, 2000.
- Hastie, Tibshirani, Friedman: The Elements of Statistical Learning, Springer, 2001.
- L. Devroye, L. Gyorfi, and G. Lugosi: A probabilistic theory of pattern recognition. Springer, New York, 1996

## Web Acknowledgements

The web-page code is based (with modifications) on the one of the course on Machine Learning (Fall Semester 2013; Prof. A. Krause).