The mathematics of change and optimization. Calculus is what makes neural networks learn — every weight update, every gradient, every backpropagation step lives here.
Every time a neural network learns, calculus runs underneath. The model asks: how wrong am I, and which direction should I move to be less wrong? Calculus answers both questions.
Loss = f(weights)
Goal → minimize Loss
Tool → Derivatives + Gradients + Optimization
Measure Error
L = loss(ŷ, y)
Sensitivity
∂L/∂wᵢ
Update Direction
-∇L
Converge
w* = argmin L
02 · Basics
Functions & Graphs
A function maps every input to exactly one output. In ML, functions model the relationship between data and predictions.
A function is continuous at x = a if it has no jumps, holes, or breaks. Most loss functions in ML are continuous — gradient descent can work smoothly.
Condition 1
f(a) exists
Condition 2
lim(x→a) f(x) exists
Condition 3
Both are equal
ML Note
MSE, Cross-entropy ✓
05 · Core Calculus
Derivatives — Intuition
A derivative measures the instantaneous rate of change. Geometrically it's the slope of the tangent line — in ML it tells us how fast the loss changes when we nudge a weight.
Formal Definition
f'(x) = lim(h→0) [f(x+h) − f(x)] / h
The slope at a single point
f(x) = x²
f'(x) = 2x
At x = 3 → slope = 6 (steep, rising)
At x = 0 → slope = 0 (flat, minimum!)
AI Intuition
∂Loss/∂weight = "if I increase this weight by a tiny amount, how much does the loss change?"
Negative derivative → decreasing loss → good direction to move!
06 · Differentiation Rules
Rules of Differentiation
Power Rule
d/dx xⁿ = n·xⁿ⁻¹
Constant Rule
d/dx c = 0
Sum Rule
d/dx(f+g) = f'+g'
Product Rule
d/dx(fg) = f'g + fg'
Quotient Rule
d/dx(f/g) = (f'g−fg')/g²
Exponential
d/dx eˣ = eˣ
Logarithm
d/dx ln(x) = 1/x
Sigmoid
σ'(x) = σ(x)(1−σ(x))
07 · Higher-Order Derivatives
Higher-Order Derivatives
The second derivative measures curvature — how the slope itself is changing. This lets us classify critical points.
f(x) = x²
f'(x) = 2x → first derivative (slope)
f''(x) = 2 → second derivative (curvature)
f''(x) > 0 → concave up → LOCAL MINIMUM ✓
f''(x) < 0 → concave down → LOCAL MAXIMUM ✓
f''(x) = 0 → inflection point → inconclusive
08 · Multivariable
Partial Derivatives
Neural networks have millions of weights. Partial derivatives let us measure how the loss changes with respect to each individual weight, holding all others constant.
f(x, y) = x² + 3xy + y²
∂f/∂x = 2x + 3y (treat y as constant)
∂f/∂y = 3x + 2y (treat x as constant)
AI Intuition
With 1 million weights: compute ∂L/∂w₁, ∂L/∂w₂, …, ∂L/∂w₁₀₀₀₀₀₀
Each tells us: "how much does this single weight affect the loss?"
09 · Gradient
Gradients & Directional Derivatives
The gradient packs all partial derivatives into a single vector — pointing in the direction of steepest increase. Negating it gives the direction of steepest descent.
Gradient Vector
∇f = [∂f/∂x₁, ∂f/∂x₂, …, ∂f/∂xₙ]
A vector in parameter space. Length encodes magnitude of change.
10 · The Key Rule
Chain Rule — Backpropagation Core
The most important rule in deep learning. Backpropagation is nothing more than the chain rule applied repeatedly through each layer.
Chain Rule
If y = f(g(x)), then dy/dx = f'(g(x)) · g'(x)
Compose derivatives layer by layer, from output back to input
Output: ŷ = f₃(f₂(f₁(x)))
∂L/∂w₁ = ∂L/∂ŷ · ∂ŷ/∂z₃ · ∂z₃/∂z₂ · ∂z₂/∂w₁
Each term: how does this layer's output
change w.r.t. the previous layer?
Why This Matters
Without the chain rule, we couldn't train deep networks. It lets us efficiently compute gradients for ALL layers in a single backward pass — O(n) not O(n²).
11 · Advanced
Jacobian & Hessian
Jacobian Matrix
J = ∂(outputs)/∂(inputs) Maps input→output sensitivity Used in: layer Jacobians
Hessian Matrix
H = ∂²f/∂xᵢ∂xⱼ Curvature in all directions Used in: Newton's method
H = ⎡ ∂²f/∂x² ∂²f/∂x∂y ⎤
⎣ ∂²f/∂y∂x ∂²f/∂y² ⎦
det(H) > 0, H₁₁ > 0 → local minimum
det(H) > 0, H₁₁ < 0 → local maximum
det(H) < 0 → saddle point
12 · Optimization
Critical Points & Classification
A critical point is any point where the gradient equals zero — the function momentarily "flattens out." The type determines whether it's a minimum, maximum, or saddle.
Critical Point Condition
∇f = 0 → f'(x) = 0 for single variable
These are candidates for minima, maxima, and saddle points
↓
Local Minimum
f''(x) > 0 f is concave up
↑
Local Maximum
f''(x) < 0 f is concave down
⇆
Saddle Point
f''(x) = 0 or det(H) < 0
13 · Training Algorithm
Gradient Descent
The engine of training. We repeatedly step in the direction of the negative gradient, nudging parameters toward a loss minimum.
Update Rule
θ ← θ − α · ∇L(θ)
θ = parameters · α = learning rate · L = loss function
B
Batch GD
Full dataset per step. Precise but slow.
S
SGD
One sample per step. Fast, noisy.
M
Mini-Batch
Best of both. Standard in practice.
14 · Convexity
Convexity
A function is convex if the line segment between any two points on the function lies above or on the curve. Convex loss functions have only one minimum — guaranteed to find it.
f is convex ⟺
f(tx + (1−t)y) ≤ tf(x) + (1−t)f(y) for all t ∈ [0,1]
Convex examples in ML:
MSE = Σ(y − ŷ)² → convex ✓
Cross-entropy loss → convex ✓
Neural network loss → non-convex (many saddles)
15–17 · Integration
Integrals & Probability
Integration accumulates change — computing areas, total probability, and expectations. In ML, integrals underpin probability distributions and Bayesian inference.
Indefinite
∫ x² dx = x³/3 + C
Definite
∫₀¹ x² dx = 1/3
Probability
∫₋∞^∞ p(x) dx = 1
Expectation
E[X] = ∫ x·p(x) dx
Multivariable
∫∫ f(x,y) dx dy
Joint Prob
∫∫ p(x,y) dx dy = 1
18 · Approximation
Taylor Series
Taylor series lets us approximate any smooth function as a polynomial around a point — how gradient descent justifies small steps.
f(x) ≈ f(a) + f'(a)(x−a) + f''(a)/2!(x−a)² + ...
For loss landscape near θ*:
L(θ) ≈ L(θ*) + ∇L(θ*)ᵀ(θ−θ*) + ½(θ−θ*)ᵀH(θ−θ*)
Linear term → gradient direction
Quadratic term → curvature (Hessian)