Description

This course begins with a brief review of machine learning and then introduces the basics of neural networks. Modern network architectures will be discussed. e.g., CNN, GNN, and transformer networks. This course focuses on the practical aspects of deep learning and its applications in computer vision and natural language processing. One-third of the course is on advanced research topics including deep structured model, generative model, foundation model, and multimodal deep learning.

Lectures

• Friday, 1:30PM - 4:20PM, COB1 Room 286

Office hours and contact information

• Instructor: Meng Tang
• Email: mtang4@umcerced.edu
• Office hours: Every Friday from 11AM to 12PM, SE2 room 279

Topics

This section gives a minimal introduction to machine learning. Students should refer to CSE 176 for a thorough introduction to machine learning.

• What is machine learning? Different types of machine learning.
• Classification and regression, clustering, logistic regression

This section introduces the basics of neural network pre deep learning era including feed-forward network, recurrent network, regularization, and optimization for neural networks.

• Logistic regression as neural network
• Feed-forward network, multi-layer perceptron
• back propagation
• Losses, Optimization, regularization, hyperparameter tuning
• Recurrent neural network (LSTM, GRU)
• Deep learning software libraries (Pytorch, Tensorflow)

This section elaborates on modern network architectures such as CNN, graph neural network, pointnet, and transformers that are popular for vision and language.

• Convolutional Neural Network
• CNN for computer vision (Classification, detection, segmentation networks)
• Visualization of CNN
• Residual neural networks
• Neural network for graph and set
• Message passing and graph neural network
• Attention and transformer
• Word embedding and language models
• Self-attention, cross-attention, transformer
• Vision transformer

This section discusses topics at the forefront of deep learning research as well as applications for multi-modal data.

• Deep structured models
• Graphical model, MRF, MAP inference
• Deep structured learning
• Deep Generative Models(VAE, GAN, diffusion model)
• Multimodal deep learning
• Foundation models for vision and language: CLIP, BERT, DETIC, Florence

Reading list

• Understanding Deep Learning by Simon J.D. Prince Published by MIT Press 2023. https://udlbook.github.io/udlbook
• Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville https://www.deeplearningbook.org
• UvA Deep Learning Tutorials https://uvadlc-notebooks.readthedocs.io/en/latest

Prerequisites

• MATH24 Linear Algebra and Differential Equations or equivalent course
• MATH32 Probability and Statistics or equivalent course
• Python programming skills

Grading

• Exams (20%)
  • Midterm 20%
• Project (50%)
  • Midterm report 10%
  • Final presentation 10%
  • Final report 30%
• Assignment (30%)
  • Assignment 1 10%
  • Assignment 2 10%
  • Assignment 3 10%