Most people searching for a deep learning cheat sheet already know what a neural network is. What they actually need is a single place to look up whether tanh or ReLU makes more sense for their hidden layer, why their training loss is diverging, or what the difference between a residual connection and a skip connection actually is at the implementation level. That's what this is.
This reference covers the concepts you'll reach for repeatedly: activation functions, loss functions, major architectures, regularization techniques, and optimization. Where a formula matters, it's included. Where the distinction is subtle, it's called out directly.
Deep Learning Cheat Sheet: Core Building Blocks
Every deep learning system is built from the same handful of components. The choices you make here—activation function, initialization, normalization—compound through every layer of training.
Activation Functions
- ReLU (Rectified Linear Unit):
f(x) = max(0, x). The default for hidden layers in most feedforward and convolutional networks. Fast to compute, avoids vanishing gradients in the positive region. Downside: "dying ReLU" problem where neurons output zero permanently if weights push them into the negative side. - Leaky ReLU:
f(x) = x if x > 0, else αx(α typically 0.01). Fixes dying ReLU by allowing a small gradient in the negative region. - GELU (Gaussian Error Linear Unit): Used in Transformers (BERT, GPT). Smoother than ReLU; approximated as
x · Φ(x)where Φ is the standard normal CDF. - Sigmoid:
σ(x) = 1 / (1 + e⁻ˣ). Output range (0, 1). Used in binary classification output layers and gates (LSTM, GRU). Avoid in hidden layers—saturates and causes vanishing gradients. - Tanh: Output range (-1, 1). Zero-centered (unlike sigmoid), which helps gradient flow. Still saturates; mostly used in RNNs and gates.
- Softmax: Converts a vector of logits to a probability distribution. Standard final layer for multi-class classification.
Loss Functions
- Binary Cross-Entropy:
-[y·log(ŷ) + (1-y)·log(1-ŷ)]. Binary classification tasks. - Categorical Cross-Entropy:
-Σ yᵢ·log(ŷᵢ). Multi-class classification with one-hot labels. - MSE (Mean Squared Error):
(1/n)·Σ(y - ŷ)². Regression. Sensitive to outliers because errors are squared. - MAE (Mean Absolute Error):
(1/n)·Σ|y - ŷ|. More robust to outliers than MSE; gradients are constant, which can cause instability near minima. - Huber Loss: MSE for small errors, MAE for large ones. Best of both in regression when outliers are present.
- KL Divergence:
Σ p(x)·log(p(x)/q(x)). Measures how one distribution differs from a reference. Core to VAEs and some GAN objectives.
Weight Initialization
- Xavier/Glorot: Variance =
2 / (n_in + n_out). Designed for tanh/sigmoid activations. - He (Kaiming): Variance =
2 / n_in. Designed for ReLU. Standard for most modern feedforward and convolutional networks. - Zero initialization: Never use for weights—all neurons become symmetric and learn the same features. Biases can be zero-initialized.
Architecture Reference on Your Deep Learning Cheat Sheet
Knowing which architecture to reach for—and why—saves more time than memorizing any formula.
Feedforward Networks (MLP)
Fully connected layers; each neuron connects to every neuron in the next layer. Good baseline for tabular data. Struggles with spatial or sequential structure. Use when you don't have a clear spatial or temporal structure in your data.
Convolutional Neural Networks (CNN)
Parameter sharing via filters makes CNNs efficient for grid-structured data (images, audio spectrograms). Key concepts:
- Receptive field: The input region a neuron "sees." Grows with depth.
- Pooling: Max pooling downsamples and introduces translation invariance. Average pooling preserves more spatial information.
- Depthwise separable convolutions: Split spatial and channel-wise convolutions. Used in MobileNet—reduces parameter count significantly.
- Residual connections (ResNet):
output = F(x) + x. Lets gradients flow directly, enabling very deep networks (100+ layers) without degradation.
Recurrent Neural Networks (RNN/LSTM/GRU)
Designed for sequential data where order matters. Vanilla RNNs fail with long sequences due to vanishing gradients.
- LSTM: Three gates (input, forget, output) plus a cell state. The forget gate decides what to drop from memory. Handles sequences up to ~100-200 steps reliably.
- GRU: Simplified LSTM with two gates (reset, update). Fewer parameters, similar performance on many tasks, trains faster.
Transformers
Attention mechanism replaces recurrence entirely. Key formula: Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V. Scales quadratically with sequence length—the core bottleneck for long contexts.
- Encoder-only (BERT): Good for classification, named entity recognition, embeddings.
- Decoder-only (GPT): Autoregressive generation.
- Encoder-decoder (T5, original Transformer): Translation, summarization, seq2seq tasks.
Generative Architectures
- VAE: Learns a compressed latent space. Loss = reconstruction loss + KL divergence. Good for structured generation and interpolation.
- GAN: Generator and discriminator trained adversarially. Training instability is the main practical challenge (mode collapse, vanishing gradients in discriminator).
- Diffusion Models: Gradually denoise from Gaussian noise. More stable training than GANs; slower inference. Current state of the art in image synthesis.
Optimization and Regularization
Optimizers
- SGD with momentum: Simple, often generalizes better than adaptive methods on image tasks. Requires careful learning rate tuning.
- Adam: Adaptive learning rates per parameter.
m̂ₜ / (√v̂ₜ + ε)update rule. Fast convergence, default for most NLP/Transformer work. Can generalize slightly worse than SGD on some vision benchmarks. - AdamW: Adam with decoupled weight decay. Fixes a regularization bug in standard Adam. Preferred for Transformer training.
- Learning rate schedulers: Cosine annealing and warmup-then-decay are standard for Transformers. Step decay for CNNs.
Regularization Techniques
- Dropout: Randomly zeros activations during training (rate typically 0.1–0.5). Reduces co-adaptation. Less common in CNNs (batch norm fills the role); common in Transformers.
- Batch Normalization: Normalizes activations across the batch. Stabilizes training, allows higher learning rates. Works poorly with small batch sizes.
- Layer Normalization: Normalizes across features within a single sample. Standard in Transformers—doesn't depend on batch size.
- L2 Regularization (Weight Decay): Penalizes large weights. Encourages simpler models.
- Early Stopping: Halt training when validation loss stops improving. Simple and effective.
- Data Augmentation: Random crops, flips, color jitter for images; back-translation or synonym replacement for text. Increases effective dataset size without collecting new data.
Common Diagnostics
- Training loss drops but validation loss stagnates: Overfitting. Add dropout, weight decay, or data augmentation.
- Both losses plateau early: Learning rate too low, or model is underfitting—try a larger model or more training.
- Loss explodes: Learning rate too high, or missing gradient clipping. For RNNs and Transformers: clip gradients at norm 1.0.
- NaN in loss: Check for log(0) in your loss function, extreme learning rates, or missing normalization.
Top Courses for Deep Learning
Reading a cheat sheet gets you oriented. Actually training models requires working through structured material. These courses have the best signal-to-noise ratio based on ratings and practical depth.
Neural Networks and Deep Learning
Andrew Ng's foundational Coursera course (rated 9.8/10) covers the math behind backpropagation, vectorization, and activation functions clearly—this is the course that explains why the formulas in this cheat sheet look the way they do, not just that they work.
Deep Learning: All Models Explained for Beginners
Rated 8.8/10 on Udemy, this course is particularly useful if you're working through the architecture section above and want code walkthroughs alongside the theory—it covers MLPs, CNNs, RNNs, and attention mechanisms in one place.
Deep Learning for Computer Vision
Coursera course rated 8.7/10, focused on applying CNNs to real image tasks. If computer vision is your target domain, this is more targeted than a general deep learning course—covers transfer learning, object detection, and segmentation architectures.
Deep Learning Methods for Healthcare
Rated 8.7/10 on Coursera—relevant if you're applying deep learning to medical imaging, clinical NLP, or time series from sensors. The domain constraints (data scarcity, interpretability requirements) are handled specifically, which general courses skip over.
Generative AI Deep Research: Strategic AI Edge for Leaders
Rated 8.7/10 on Coursera—aimed at practitioners who need to apply large generative models rather than build them from scratch. Bridges the gap between the architecture reference above and production deployment decisions.
FAQ
What should a deep learning cheat sheet actually include?
At minimum: the activation functions you'll choose between (ReLU, sigmoid, softmax, GELU), the loss functions for classification and regression, the major architecture types (CNN, RNN, Transformer), the optimizers (SGD, Adam, AdamW), and the regularization techniques (dropout, batch norm, weight decay). A good cheat sheet is a lookup tool, not a tutorial—dense reference over explanation.
Is deep learning the same as machine learning?
Deep learning is a subset of machine learning. Machine learning covers a broader set of techniques—including decision trees, SVMs, gradient boosting, and linear models. Deep learning specifically refers to neural networks with multiple layers (typically more than two hidden layers). For tabular data, gradient boosting (XGBoost, LightGBM) frequently outperforms deep learning. For images, text, and audio, deep learning is the dominant approach.
What math do I need before studying deep learning?
Linear algebra (matrix multiplication, dot products, eigendecomposition), calculus (partial derivatives, chain rule—backpropagation is just the chain rule applied repeatedly), and probability (distributions, Bayes' theorem, cross-entropy). You don't need graduate-level expertise in any of these—a solid undergraduate understanding is sufficient to start building models and refine the math as you go.
What's the difference between parameters and hyperparameters?
Parameters are learned during training: weights and biases in each layer. Hyperparameters are set before training and control the learning process: learning rate, batch size, number of layers, dropout rate, weight decay. Hyperparameter tuning (grid search, random search, Bayesian optimization) is the process of finding values that minimize validation loss.
When should I use a pre-trained model instead of training from scratch?
Almost always. For images, start with ImageNet-pretrained weights (ResNet, EfficientNet, ViT) and fine-tune. For text, start with a pretrained language model (BERT, RoBERTa, GPT-2). Training from scratch requires large datasets and significant compute—fine-tuning a pretrained model can match or beat scratch training with 10–100x less data. Exceptions: highly domain-specific inputs (e.g., certain medical imaging modalities) where pretrained features don't transfer well.
What's the best framework for deep learning in 2026?
PyTorch dominates research and is increasingly used in production. TensorFlow/Keras remains common in production pipelines built before 2022, and JAX is growing in research contexts that require hardware-level control (Google TPUs, large-scale distributed training). For most practitioners, PyTorch is the right default—the ecosystem (Hugging Face, Lightning, torchvision) is comprehensive and the debugging experience is better than static graph frameworks.
Bottom Line
A deep learning cheat sheet is only useful if you already have enough context to know what you're looking up. If activation functions and loss functions are mostly clear but you're fuzzy on when to use them, start with the Neural Networks and Deep Learning course—it's the most direct path to making this reference actually click. If you have the fundamentals and want to go deeper in a specific domain (vision, healthcare, generative AI), the domain-specific courses above will get you there faster than another general overview.
The concepts above—activation functions, architectures, optimization, regularization—don't change quickly. What changes is which architectures are dominant (Transformers ate most of the space in the last four years) and what tooling looks like. The math stays stable. Learn it once, and the cheat sheet stops being a crutch and starts being a quick sanity check.