Stage 4: Deep Learning and Neural Networks

Stanford CS336 course, systematically explaining how to build a large language model from scratch, covering the complete process from data processing, Transformer architecture, model training, GPU optimization, parallel computing to RLHF alignment.

Start Learning AI →
LanguageModelTransformerStanfordYouTubeVideoFreeEnglish

Stanford CS336: Language Modeling from Scratch | Spring 2025

Course Overview

Course Title: CS336 - Language Modeling from Scratch
Offering Term: Spring 2025
Offered by: Stanford Online
Course Format: Complete Video Lecture Series (17 lectures)
Release Date: July 8, 2025

Course Description

Language models are the cornerstone of modern Natural Language Processing (NLP) applications and have ushered in a new paradigm: a single general-purpose system capable of handling various downstream tasks. As the fields of Artificial Intelligence (AI), Machine Learning (ML), and Natural Language Processing continue to evolve, a deep understanding of language models has become crucial for scientists and engineers alike.

This course aims to provide students with a comprehensive understanding of language models by guiding them through the entire process of developing their own language model. Drawing inspiration from the idea of building an entire operating system from scratch, this course will take students through every aspect of language model creation, including:

  • Data Collection and Cleaning (for pre-training)
  • Transformer Model Construction
  • Model Training
  • Pre-deployment Evaluation

Course Information

Course Syllabus

Lecture 1: Overview and Tokenization (1:18:59)

  • Course Overview
  • Introduction to Tokenization
  • Views: 250k+

Lecture 2: PyTorch, Resource Accounting (1:19:22)

  • Using the PyTorch Framework
  • Resource Accounting
  • Views: 87k+

Lecture 3: Architectures, Hyperparameters (1:27:03)

  • Model Architecture Design
  • Hyperparameter Tuning
  • Views: 65k+

Lecture 4: Mixture of Experts (1:22:04)

  • Mixture of Experts Models
  • Views: 46k+

Lecture 5: GPUs (1:14:21)

  • GPU Computing Principles and Applications
  • Views: 39k+

Lecture 6: Kernels, Triton (1:20:22)

  • Kernel Optimization
  • Triton Framework
  • Views: 26k+

Lecture 7: Parallelism 1 (1:24:42)

  • Parallel Computing Techniques (Part 1)
  • Views: 24k+

Lecture 8: Parallelism 2 (1:15:18)

  • Parallel Computing Techniques (Part 2)
  • Views: 15k+

Lecture 9: Scaling Laws 1 (1:05:18)

  • Scaling Laws (Part 1)
  • Views: 18k+

Lecture 10: Inference (1:22:52)

  • Inference Optimization
  • Views: 19k+

Lecture 11: Scaling Laws 2 (1:18:13)

  • Scaling Laws (Part 2)
  • Views: 13k+

Lecture 12: Evaluation (1:20:48)

  • Model Evaluation Methods
  • Views: 13k+

Lecture 13: Data 1 (1:19:06)

  • Data Processing (Part 1)
  • Views: 14k+

Lecture 14: Data 2 (1:19:12)

  • Data Processing (Part 2)
  • Views: 12k+

Lecture 15: Alignment - SFT/RLHF (1:14:51)

  • Alignment Techniques
  • Supervised Fine-Tuning (SFT)
  • Reinforcement Learning from Human Feedback (RLHF)
  • Views: 19k+

Lecture 16: Alignment - RL 1 (1:20:32)

  • Alignment - Reinforcement Learning (Part 1)
  • Views: 19k+

Lecture 17: Alignment - RL 2 (1:16:09)

  • Alignment - Reinforcement Learning (Part 2)
  • Views: 16k+

Course Features

  1. Comprehensive Scope: Covers the complete language model development pipeline, from data preparation to model deployment.
  2. Practice-Oriented: Emphasizes hands-on practice, with students building their own language models.
  3. In-depth Technical Content: Covers advanced topics such as GPU optimization, parallel computing, and Triton.
  4. Cutting-edge Content: Includes the latest alignment techniques (RLHF) and research on scaling laws.
  5. Engineering Practice: Focuses on engineering issues like resource accounting and performance optimization.

Target Audience

  • Researchers seeking a deep understanding of large language model mechanisms.
  • Engineers who want to build language models from scratch.
  • Students with a foundational understanding of NLP and deep learning.
  • Scientists and practitioners in the AI/ML fields.

Prerequisites

  • Solid programming foundation (Python)
  • Basic knowledge of deep learning
  • Familiarity with fundamental neural network concepts
  • Familiarity with basic machine learning theory

Learning Resources

Summary

This is a highly valuable course for learners who genuinely want to understand and master language model technology. Through systematic study, students will be able to independently build, train, and deploy their own language models, gaining a deep understanding of the most cutting-edge NLP technologies available today.