Statistical Learning Theory, Spring Semester 2024

Instructors

Prof. Dr. Joachim M. Buhmann

Assistants

Vignesh Ram Somnath
Dr. Alina Dubatovka
Evgenii Bykovetc
Dr. Fabian Laumer
Ivan Ovinnikov
João Lourenço Borges Sá Carvalho
Patrik Okanovic
Robin Geyer
Xia Li
Yilmazcan Özyurt

News

  • The exam will be held on June 3 2024, from 9 AM - 12 PM in rooms ETA F5 & ETF C1.

General Information

This is the last offering of the Statistical Learning Theory course.

The ETHZ Course Catalogue information can be found here.

The course covers advanced methods of statistical learning. The fundamentals of Machine Learning as presented in the course "Introduction to Machine Learning" and "Advanced Machine Learning" are expanded and, in particular, the following topics are discussed:

  • Variational methods and optimization. We consider optimization approaches for problems where the optimizer is a probability distribution. We will discuss concepts like maximum entropy, information bottleneck, and deterministic annealing.
  • Clustering. This is the problem of sorting data into groups without using training samples. We discuss alternative notions of "similarity" between data points and adequate optimization procedures.
  • Model selection and validation. This refers to the question of how complex the chosen model should be. In particular, we present an information theoretic approach for model validation.
  • Statistical physics models. We discuss approaches for approximately optimizing large systems, which originate in statistical physics (free energy minimization applied to spin glasses and other models). We also study sampling methods based on these models.

Please use Moodle for questions regarding course material, organization and projects.

Time and Place

Type Time Place
Lectures Mon 10:15-12:00 HG E 7
Tue 17:15-18:00 HG G 5
Exercises Mon 16:15-18:00 HG G 3

Lecture Notes

The latest version of the course notes can be found here.

An older version of the same script can be found at here. It's no longer maintained, but it contains useful notes for some chapters not covered yet in the latest notes.

Lectures

Date and Topics Lecture Slides Recording Links
Feb 19

Introduction
Motivation
Introduction
Probability Basics
Recording 1
Recording 2
Feb 26

Maximum Entropy
Entropy Recording 1
Recording 2
Mar 4

Maximum Entropy
Sampling
Maximum Entropy Inference Recording 1
Recording 2
Mar 11

Deterministic Annealing
Maximum Entropy Training Recording 1
Recording 2
Mar 18

Clustering
Least Angle Clustering

Tutorials

Date Tutorial Recording Links Exercises
Februrary 26 Calculus Recap
Functional Derivatives
Recording Exercise 1
Solution 1
March 4 Tutorial taught on blackboard Recording Exercise 2
Solution 2
March 11 Sampling Exercise 3
Solution 3
March 18 Exercise 4
Solution 4

Past written Exams

2018 [Exam] [Solution]
2019 [Exam] [Solution]
2020 [Exam (with solution)]

For this year, the exam will be be an end-of-semester examination. The exam will be held on June 3 2024, from 9 AM - 12 PM in rooms ETA F5 and ETF C1.

Projects

Projects are coding exercises that concern the implementation of an algorithm taught in the lecture/exercise class.

There will be four coding exercises, with a time span of approximately two weeks per coding exercise. Each one of them will be graded as not passed or with a passing grade ranging from 4 to 6.

In order to be admitted to the exam the student has to pass (i.e. a grade of 4) in 3 of the 4 projects, and the final grade for the whole class is the weighted average 0.7 exam + 0.3 project. The coding exercises will be provided and submitted via moodle.

Project Release Date Project Due Date Topic Moodle Link
March 4 March 18 Sampling and Annealing Coding Exercise 1
March 25 April 15 Histogram Clustering
April 22 May 6 Constant Shift Embeddings
May 13 May 27 Model Validation

Other Resources

  • Duda, Hart, Stork: Pattern Classification, Wiley Interscience, 2000.
  • Hastie, Tibshirani, Friedman: The Elements of Statistical Learning, Springer, 2001.
  • L. Devroye, L. Gyorfi, and G. Lugosi: A probabilistic theory of pattern recognition. Springer, New York, 1996

Web Acknowledgements

The web-page code is based (with modifications) on the one of the course on Machine Learning (Fall Semester 2013; Prof. A. Krause).