Dive into Deep Learning (D2L) Course Detailed Introduction

Course Overview

Dive into Deep Learning (D2L) is an open-source deep learning textbook designed to make deep learning easy to understand and learn. It combines concepts, context, and code to provide readers with a comprehensive resource for learning deep learning.

Course Background

A few years ago, it was rare for deep learning scientists to develop intelligent products and services in large companies and startups. At that time, machine learning was not daily headline news, and our parents didn't know what machine learning was. Machine learning was then an academic blue-sky discipline, with its industrial significance limited to a few practical applications, including speech recognition and computer vision.

However, in just a few years, deep learning has astonished the world, driving rapid advancements in multiple fields such as computer vision, natural language processing, automatic speech recognition, reinforcement learning, and biomedical informatics. The success of deep learning has even catalyzed the development of theoretical machine learning and statistics.

Textbook Features

1. Diverse Media Integration

• Combination of Code, Math, and HTML: Integrates code, mathematical formulas, and web content on one platform.
• Executable Code: Most chapters include executable code examples.
• Real-time Updates: Supports rapid updates and community contributions via GitHub.

2. Learning Approach: Learning by Doing

• Just-in-Time Concept Learning: Concepts are taught only when needed to solve practical problems.
• Practice First: Allows students to experience the satisfaction of training their first model before worrying about more esoteric concepts.
• Real-world Datasets: Each subsequent chapter provides several self-contained working examples using real-world datasets.

3. Multi-Framework Support

The textbook supports implementations across multiple deep learning frameworks:

• PyTorch (primary framework)
• TensorFlow
• MXNet
• JAX

Content Structure

The textbook can be divided into three main parts:

Part I: Foundations and Preliminaries

• Chapter 1: Introduction to Deep Learning
• Chapter 2: Preliminaries (data storage and manipulation, numerical operations, etc.)
• Chapter 3: Linear Regression
• Chapter 5: Multilayer Perceptrons
• Covers fundamental concepts such as regression and classification, linear models, multilayer perceptrons, overfitting, and regularization.

Part II: Modern Deep Learning Techniques

• Chapter 6: Key Computational Components of Deep Learning Systems
• Chapters 7-8: Convolutional Neural Networks (CNN)
• Chapters 9-10: Recurrent Neural Networks (RNN)
• Chapter 11: Attention Mechanisms and Transformers
• Covers the most powerful and versatile tools widely used by modern deep learning practitioners.

Part III: Scalability, Efficiency, and Applications (Available Online)

• Chapter 12: Optimization Algorithms
• Chapter 13: Computational Performance
• Chapter 14: Computer Vision Applications
• Chapters 15-16: Language Representation Model Pre-training and Natural Language Processing Applications

Code Implementation Features

Core Package and Dependencies

The textbook uses the d2l package to avoid unnecessary repetition. This package is lightweight and only requires the following dependencies:

#@save
import collections
import hashlib
import inspect
import math
import os
import random
import re
import shutil
import sys
import tarfile
import time
import zipfile
from collections import defaultdict
import pandas as pd
import requests
from IPython import display
from matplotlib import pyplot as plt
from matplotlib_inline import backend_inline
d2l = sys.modules[__name__]

PyTorch Implementation Dependencies

#@save
import numpy as np
import torch
import torchvision
from PIL import Image
from scipy.spatial import distance_matrix
from torch import nn
from torch.nn import functional as F
from torchvision import transforms

TensorFlow Implementation Dependencies

#@save
import numpy as np
import tensorflow as tf

JAX Implementation Dependencies

#@save
from dataclasses import field
from functools import partial
from types import FunctionType
from typing import Any
import flax
import jax
import numpy as np
import optax
import tensorflow as tf
import tensorflow_datasets as tfds
from flax import linen as nn
from flax.training import train_state
from jax import grad
from jax import numpy as jnp
from jax import vmap

Target Audience

This textbook is intended for:

• Students (undergraduate or graduate)
• Engineers
• Researchers

who are seeking a solid grasp of practical deep learning techniques.

Prerequisites

• A moderate understanding of linear algebra
• Basic calculus
• Basic probability theory
• Basic Python programming

Learning Resources

Online Resources

• Official Website: D2L.ai
• GitHub Repository: d2l-ai/d2l-en
• Discussion Forum: discuss.d2l.ai
• Other Language Versions: Other Language Versions

Access Methods

• Executable code format
• Physical book
• Downloadable PDF
• Online website version

All notebooks are freely available for download on the D2L.ai website and GitHub.

Textbook Advantages

1. Free and Open: Available to everyone at no cost.
2. Technical Depth: Provides the technical depth required to become an applied machine learning scientist.
3. Runnable Code: Demonstrates how to solve problems in practice.
4. Rapid Updates: Supports quick updates from authors and the community.
5. Community Support: Accompanied by an interactive forum for technical discussions and Q&A.

Innovative Features

• Possibly the first book published using an integrated workflow.
• Combines textbook-quality exposition with clean, runnable code from practical tutorials.
• Uses GitHub to share source code, with Jupyter notebooks mixing code, equations, and text.
• Employs Sphinx as the rendering engine and Discourse as the discussion platform.

Learning Suggestions

1. Register for a discussion forum account.
2. Install Python on your computer.
3. Seek help and discuss via the forum links at the bottom of each chapter.
4. Learn by doing, making full use of the executable code.
5. Participate in community discussions and interact with the authors and the broader community.

This textbook represents a significant milestone in deep learning education, providing learners with a comprehensive, practical, and easily accessible resource for studying deep learning.