Statistical Learning Theory, Spring Semester 2024
Instructors
Prof. Dr. Joachim M. BuhmannAssistants
Vignesh Ram SomnathDr. Alina Dubatovka
Evgenii Bykovetc
Dr. Fabian Laumer
Ivan Ovinnikov
João Lourenço Borges Sá Carvalho
Patrik Okanovic
Robin Geyer
Xia Li
Yilmazcan Özyurt
News
- The exam will be held on June 3 2024, from 9 AM - 12 PM in rooms ETA F5 & ETF C1.
General Information
This is the last offering of the Statistical Learning Theory course.
The ETHZ Course Catalogue information can be found here.
The course covers advanced methods of statistical learning. The fundamentals of Machine Learning as presented in the course "Introduction to Machine Learning" and "Advanced Machine Learning" are expanded and, in particular, the following topics are discussed:
- Variational methods and optimization. We consider optimization approaches for problems where the optimizer is a probability distribution. We will discuss concepts like maximum entropy, information bottleneck, and deterministic annealing.
- Clustering. This is the problem of sorting data into groups without using training samples. We discuss alternative notions of "similarity" between data points and adequate optimization procedures.
- Model selection and validation. This refers to the question of how complex the chosen model should be. In particular, we present an information theoretic approach for model validation.
- Statistical physics models. We discuss approaches for approximately optimizing large systems, which originate in statistical physics (free energy minimization applied to spin glasses and other models). We also study sampling methods based on these models.
Please use Moodle for questions regarding course material, organization and projects.
Time and Place
Type | Time | Place |
---|---|---|
Lectures | Mon 10:15-12:00 | HG E 7 |
Tue 17:15-18:00 | HG G 5 | |
Exercises | Mon 16:15-18:00 | HG G 3 |
Lecture Notes
The latest version of the course notes can be found here.
An older version of the same script can be found at here. It's no longer maintained, but it contains useful notes for some chapters not covered yet in the latest notes.
Lectures
Date and Topics | Lecture Slides | Recording Links | |
---|---|---|---|
Feb 19
Introduction |
Motivation
Introduction Probability Basics |
Recording 1
Recording 2 |
|
Feb 26
Maximum Entropy |
Entropy |
Recording 1
Recording 2 |
|
Mar 4
Maximum Entropy Sampling |
Maximum Entropy Inference |
Recording 1
Recording 2 |
|
Mar 11
Deterministic Annealing |
Maximum Entropy Training |
Recording 1
Recording 2 |
|
Mar 18
Clustering |
Least Angle Clustering |
Tutorials
Date | Tutorial | Recording Links | Exercises |
---|---|---|---|
Februrary 26 |
Calculus Recap
Functional Derivatives |
Recording |
Exercise 1
Solution 1 |
March 4 | Tutorial taught on blackboard | Recording |
Exercise 2
Solution 2 |
March 11 | Sampling |
Exercise 3
Solution 3 |
|
March 18 |
Exercise 4
Solution 4 |
Past written Exams
2018 [Exam] [Solution]
2019 [Exam] [Solution]
2020 [Exam (with solution)]
Projects
Projects are coding exercises that concern the implementation of an algorithm taught in the lecture/exercise class.
There will be four coding exercises, with a time span of approximately two weeks per coding exercise. Each one of them will be graded as not passed or with a passing grade ranging from 4 to 6.
In order to be admitted to the exam the student has to pass (i.e. a grade of 4) in 3 of the 4 projects, and the final grade for the whole class is the weighted average 0.7 exam + 0.3 project. The coding exercises will be provided and submitted via moodle.
Project Release Date | Project Due Date | Topic | Moodle Link |
---|---|---|---|
March 4 | March 18 | Sampling and Annealing | Coding Exercise 1 |
March 25 | April 15 | Histogram Clustering | |
April 22 | May 6 | Constant Shift Embeddings | |
May 13 | May 27 | Model Validation |
Other Resources
- Duda, Hart, Stork: Pattern Classification, Wiley Interscience, 2000.
- Hastie, Tibshirani, Friedman: The Elements of Statistical Learning, Springer, 2001.
- L. Devroye, L. Gyorfi, and G. Lugosi: A probabilistic theory of pattern recognition. Springer, New York, 1996
Web Acknowledgements
The web-page code is based (with modifications) on the one of the course on Machine Learning (Fall Semester 2013; Prof. A. Krause).