Taylor Expansion: The Unsung Hero of Modern AI

Taylor expansion is one of the most widely taught topics in calculus. Every year, millions of college students learn to compute Taylor series, memorize the formulas, and solve problems to ace their exams. But here's the question: beyond all those visualizations with wiggling curves and polynomial approximations, do we actually understand what Taylor expansion is really about?

More importantly, do we realize that this seemingly abstract mathematical concept is quietly powering every major breakthrough in artificial intelligence? From the gradient descent algorithms training ChatGPT to the optimization techniques behind self-driving cars, Taylor expansion is the unsung mathematical hero working behind the scenes.

Today, we're going to uncover the real story of Taylor expansion, not just as a calculus exercise, but as the fundamental tool that makes modern AI possible.

What is Taylor series?

Naturally, this should be our first question to answer. Here's the formula:

$$ f(x) = \sum_{n=0}^{\infty} \frac{f ^{(n)}(a)} { n!} (x-a)^n $$

This expands to:

$$ f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \frac{f'''(a)}{3!}(x-a)^3 + \cdots $$

Already daunting, right? But, it can get worse. For real-world applications, we need the multivariate version:

$$ f(\mathbf{x}) = \sum_{|\boldsymbol{\alpha}|=0}^{\infty} \frac{1}{\boldsymbol{\alpha}!} \frac{\partial^{|\boldsymbol{\alpha}|} f(\mathbf{a})}{\partial \mathbf{x}^{\boldsymbol{\alpha}}} (\mathbf{x} - \mathbf{a})^{\boldsymbol{\alpha}} $$

$$ f(\mathbf{x}) = f(\mathbf{a}) + \nabla f(\mathbf{a})^T (\mathbf{x} - \mathbf{a}) + \frac{1}{2} (\mathbf{x} - \mathbf{a})^T \mathbf{H}_f(\mathbf{a}) (\mathbf{x} - \mathbf{a}) + \mathcal{O}(|\mathbf{x} - \mathbf{a}|^3) $$

Frustrated? Don't worry. I'm not going to derive these formulas. 3Blue1Brown does that awesomely and beautifully in this video.

But here's what you might still be missing even after watching: Taylor series is fundamentally about local approximation (the video also emphasizes this point repeatedly, as seen in the $\ln(x)$ example). However, those overwhelmingly beautiful global examples like $\sin(x)$ and $e^x$ deceive people, even though they are special cases where the local approximation miraculously works everywhere. These are called analytic functions with infinite radius of convergence $R = \infty$.

Don't let these special cases fool you. For our journey into AI and optimization, remembering that Taylor series is fundamentally about local approximation will be the key to understanding everything that follows.

OK, so Taylor series is for local approximation. But where and why do we use it?

Here's the key insight: Taylor series assumes we already know what $f(x)$ is and understand its complete structure. The problem isn't that the function is unknown. It's that the function is too complex, computationally expensive, or otherwise intractable to work with directly.

So we use Taylor "expansion" to find a locally approximated form around $x = a$, making it much easier to calculate values near that point.

But what exactly are "series" and "expansion"?

The "series" part means we can approximate the function with arbitrarily high fidelity by adding more and more terms. The "expansion" part means we're expanding the function into simpler polynomial pieces. Basically, you choose how accurate you need to be. More terms = higher accuracy, but also more computational cost. You might only need a 1st-degree approximation for your application, or maybe you need 2nd or 3rd degree. It's your call.

1st-order approximation (linear) around $ x = a$:
$$ f(x) = f(a) + f'(a)(x-a) $$
$$ f(\mathbf{x}) = f(\mathbf{a}) + \nabla f(\mathbf{a})^T (\mathbf{x} - \mathbf{a}) $$

2nd-order approximation (quadratic) around $ \mathbf{x} = \mathbf{a} $:
$$ f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 $$
$$ f(\mathbf{x}) = f(\mathbf{a}) + \nabla f(\mathbf{a})^T (\mathbf{x} - \mathbf{a}) + \frac{1}{2} (\mathbf{x} - \mathbf{a})^T \mathbf{H}_f(\mathbf{a}) (\mathbf{x} - \mathbf{a}) $$

You just told me Taylor expansion assumes we KNOW $f(x)$, but, I hear that AI or ML is about finding UNKNOWN functions. So how can Taylor expansion power AI? That's contradictory!

Estimating unknown functions

The concept of function is crazily attractive and universal. We use it everywhere, often without realizing it. A function in mathematics is really just a system - something that takes inputs and produces outputs according to some rule. In physics, functions represent the laws of nature. In society, they represent how our world responds to actions and policies. In economics, they capture how markets behave.

The pattern is always the same: given some inputs that describe our current state, the function generates a new state. But here's the thing: in the real world, we rarely know what these functions actually look like, even though we want to be God, who know what the real function looks like as truth.

So here's the machine learning approach: if you want to discover some unknown function in your domain, mixing what you know with what you don't know yet, simply call it $f(\mathbf{x})$ and let the data "learn" that function through training. We're not limited by traditional mathematical or statistical approaches anymore. We can throw virtually any modeling architecture at the problem: take input $\mathbf{x}$, generate output $\mathbf{y}$. It could be a deep neural network with stacked dense layers and activations. Or it could be transformer blocks (built from multi-head self-attention, feed-forward layers, normalizations, and residual connections) like the GPT architecture powering ChatGPT. We don't need to know the true function. We just need to approximate it well enough.

According to the Universal Approximation Theorem on compact domains, a neural network with just one hidden layer of sufficient width can approximate any continuous function arbitrarily well. This works because neural networks are essentially piecewise function approximators - each neuron with its activation function creates local pieces that cover different domains of the input space, and the network learns to combine these pieces to approximate the target function. So function approximation is theoretically possible, though practical applications often need much deeper, more sophisticated architectures plus techniques like residual connections (to help gradient flow), gradient clipping, batch/layer normalization, multi-phase training, warmup schedules, learning rate adjustment, etc.

I got your point. But why am I talking about all these details?

Let me clarify what I'm really trying to say:

• ML/AI is fundamentally about function approximation through a "learning" process. (To be more precise, AI has a much wider scope than ML, but nowadays people use AI and ML interchangeably... 🤷‍♂️)
• We use Taylor expansion in this learning process.

Is that clear?

Thanks for your clarification. But, you said, Taylor expansion assumes we know the underlying function already. Also, ML says we don't know the underlying function so we need to learn that function. I am still confused.

Optimization Objective Function - Loss Function

To learn the underlying functions, we need a way to measure how good our current guess is. Think of it as a distance measure - how far are we from the truth?

This measure guides our search: we jump around the parameter space, following the signal until we reach a point where the measure says "you've arrived at the best place you can find" (a critical point - could be local or global minimum).

This measure is called a Loss function or Objective function, depending on your field, but they're the same concept. Let's call it $L(\boldsymbol{\theta})$ - the Loss function. This function tells us how far our current estimation is from the ground truth.

Using this signal, we iteratively try different parameter values $\boldsymbol{\theta}$, always seeking that perfect score where loss equals zero (or in practice, the minimum possible loss). It's an iterative fitting process that we actually do in our daily lives all the time.

We know this loss function $L(\boldsymbol{\theta})$ as we designed it! Even though we don't know the target function we're trying to learn, we absolutely know how to measure our current performance.

Parameters and Loss Functions

What are Parameters?

Think of parameters as the "genetic code" that determines a system's behavior. Consider a quadratic function: $f(x) = ax^2 + bx + c$. The values $\theta = [a, b, c]$ are called parameters - they define the nature and behavior of this function. Change $a$ from 2 to 0, and suddenly, you don't have a quadratic anymore, but a linear function.

Simply speaking, parameters are the knobs that control how any system behaves. When statisticians talk about "parameter estimation," they're trying to discover these hidden knobs that govern real-world phenomena.

Parameters Across Domains

Parameters are the critical pieces of information that describe the underlying target in essence, so they manifest differently across domains::

• Mathematics: (function) parameters are coefficients when function structure is already decided ($ax^2 + bx + c$)
• Machine Learning: (model) parameters are weights and biases when model architecture is already decided
• Descriptive statistics parameters - like mean, variance, skewness, kurtosis that describe any dataset

Whether you call them parameters, weights, or coefficients, the core idea remains: we're adjusting numerical knobs to make our system match the patterns we observe. The domain changes, but the fundamental concept stays the same.

Loss Functions: Measuring the Gap

If we design a model represented by parameters $\boldsymbol{\theta}$ (the model parameters we discussed, weights and biases), we can write the model as $f(\mathbf{x}; \boldsymbol{\theta})$. The loss function then becomes $L(\boldsymbol{\theta})$, which depends entirely on our choice of parameter values. Different values of $\boldsymbol{\theta}$ will make the loss decrease, increase, or ideally reach zero.

For the specific loss function structure, the most common choice is the L2 loss (least squares):

$$ L(\boldsymbol{\theta}) = \frac{1}{2}(\mathbf{y} - \hat{\mathbf{y}})^T(\mathbf{y} - \hat{\mathbf{y}}) $$

where:

$$ \hat{\mathbf{y}} = \begin{bmatrix} f(\mathbf{x}_1; \boldsymbol{\theta}) \\ f(\mathbf{x}_2; \boldsymbol{\theta}) \\ \vdots \\ f(\mathbf{x}_n; \boldsymbol{\theta}) \end{bmatrix} $$

This measures the squared difference between our predictions $f(\mathbf{x}; \boldsymbol{\theta})$ and ground truth $\mathbf{y}$.

Why squared distance instead of absolute difference? Two reasons:

1. Smooth derivatives: $ \frac{d}{d\mathbf{x}}\mathbf{x}^T\mathbf{x} = 2\mathbf{x} $ is much nicer than $\frac{d}{d\mathbf{x}}\|\mathbf{x}\|_1$, which isn't differentiable at zero
2. Emphasizes large errors: As differences grow, squared loss penalizes them exponentially more

Is L2 the only option? Absolutely not. We can definitely use L1 loss, if you want. You can also design any loss function that measures the gap between estimation and ground truth. For classification, we widely use cross-entropy loss, but this one deserves its own deep dive into information theory, entropy, and Claude Shannon's foundational work. There are also other losses such as Huber, Hinge, 0-1, etc.

Applying Taylor Expansion to Loss Functions

Now, let's apply Taylor expansion to our loss function:

$$L(\boldsymbol{\theta}) \approx L(\boldsymbol{\theta}_0) + \nabla L(\boldsymbol{\theta}_0)^T (\boldsymbol{\theta} - \boldsymbol{\theta}_0)$$

This shows how we can approximate $L(\boldsymbol{\theta})$ near $\boldsymbol{\theta} = \boldsymbol{\theta}_0$ using first-order Taylor expansion.

Wait... doesn't this look familiar? You've probably seen this formula countless times in textbooks about gradient descent:

$$\boldsymbol{\theta}_{\text{new}} = \boldsymbol{\theta}_0 - \alpha \nabla L(\boldsymbol{\theta}_0)$$

Let's connect these dots - this is the critical bridge between Taylor expansion and gradient descent that powers modern AI! 😃

The Connection:

We know our current parameters $\boldsymbol{\theta}_0$ and the loss $L(\boldsymbol{\theta}_0)$. If we move slightly to $\boldsymbol{\theta}_0 + \Delta\boldsymbol{\theta}$, Taylor expansion tells us the new loss will be approximately:

$$L(\boldsymbol{\theta}_0 + \Delta\boldsymbol{\theta}) \approx L(\boldsymbol{\theta}_0) + \nabla L(\boldsymbol{\theta}_0)^T \Delta\boldsymbol{\theta}$$

Awesome! We can estimate this new loss value without computing the expensive function - just using our local linear approximation!

Then, what should $\Delta\boldsymbol{\theta}$ be?

This is where gradient descent enters! Since $L(\boldsymbol{\theta}_0)$ is fixed, to minimize the loss we need to make the second term $\nabla L(\boldsymbol{\theta}_0)^T \Delta\boldsymbol{\theta}$ as negative as possible.

The Optimal Choice:
$$\Delta\boldsymbol{\theta} = -\alpha \nabla L(\boldsymbol{\theta}_0)$$

Why this works:

• Algebraically: Constrained optimization using Lagrange multipliers shows this is optimal within our "trust region". You try it for yourself, if you are good at solving algebraic problems. 😄
• Geometrically: The second term is a dot product of two vectors. To minimize $\mathbf{a} \cdot \mathbf{b} = |\mathbf{a}||\mathbf{b}|\cos(\theta)$, we need $\cos(\theta) = -1$, which means the vectors point in opposite directions (180°)

The Famous Formula:
$$\boldsymbol{\theta}_{\text{new}} = \boldsymbol{\theta}_0 + \Delta\boldsymbol{\theta} = \boldsymbol{\theta}_0 - \alpha \nabla L(\boldsymbol{\theta}_0)$$

Where $\alpha$ (the learning rate) is a hyperparameter controlling how big a step we take in our local approximation.

Wow! Gradient descent is just Taylor expansion telling us the optimal direction to move!

Connecting the Dots: Taylor Expansion Powers Modern AI

We've now connected Taylor expansion to gradient descent - the learning algorithm that powers modern AI/ML. At first glance, they might seem contradictory, but they're directly linked through a crucial insight:

We apply Taylor expansion to the loss function (which we know), not to the underlying function we're trying to learn.

Once this distinction is clear, everything clicks together beautifully. Ancient mathematical theorems are still alive and driving our modern AI era - we owe tremendous gratitude to the generations of mathematicians who built these foundations for us!

Going Deeper: Key Extensions and Practical Notes

1. Second-Order Methods (Newton's Method)

We can extend to second-degree Taylor approximation:
$$L(\boldsymbol{\theta}) \approx L(\boldsymbol{\theta}_0) + \nabla L(\boldsymbol{\theta}_0)^T (\boldsymbol{\theta} - \boldsymbol{\theta}_0) + \frac{1}{2}(\boldsymbol{\theta} - \boldsymbol{\theta}_0)^T \mathbf{H} (\boldsymbol{\theta} - \boldsymbol{\theta}_0)$$

Higher-order methods provide more accurate local approximations, but at the cost of increased computation. Newton's method can find optimal points instantly for well-behaved functions (like quadratics) by incorporating curvature information through the Hessian matrix $\mathbf{H}$ (second derivatives). However, computing and inverting the Hessian is computationally expensive, so first-order methods typically win in practice due to their tractability.

2. Mathematical Rigor vs. Practical Reality

In ML, we don't obsess over continuity or differentiability at every point. Unless we encounter extreme cases, we use reasonable approximations and move forward and it works remarkably well in practice! The robustness of these methods is often surpising.

3. The Power of Matrix/Vector Notation

I strongly recommend mastering matrix/vector calculus notation. Here's why:

• Multi-dimensional intuition: You develop intuition for high-dimensional spaces
• Efficient computation: Process many dimensions simultaneously (this is why we use GPUs for ML!)
• Clean implementation: Once you choose consistent notation (numerator vs denominator layout), you can: 1. Express each layer's derivatives as vectors, 2. Stack layer derivatives using chain rule, 3. Translate directly to NumPy matrix operations, 4. Handle high-dimensional tensors and batch dimensions effortlessly

Mathematical formulations can translate directly to code. In other words, no confusion, no pondering, just instant implementation from theory to practice.

4. Closed Form vs. Iterative Solutions

In mathematics, we often learn about closed-form solutions. But real-world problems aren't clean or ideal. There are countless variables, massive datasets, and sometimes no analytical solutions exist. So we embrace iterative approaches like gradient descent to find reasonably good (approximated) critical points through computational effort rather than analytical perfection.

5. The Reality of Loss Landscapes

Math textbooks start with well-behaved functions: clear convexity, continuous derivatives, easily found critical points. Differentiate twice, check curvature, identify minima. Easy!

But real loss functions are landmine fields. They're riddled with saddle points, locations where gradients vanish but aren't global optima. These create serious optimization challenges, though many techniques exist to navigate these treacherous landscapes.

6. The Numerical Challenges of Chain Rule

Chain rule works beautifully in mathematical formulas, but model training constantly battles numerical instability:

• Vanishing gradients: When each layer's gradient is small, multiplying them layer-by-layer approaches zero. Zero gradients mean no weight updates and no learning.
• Exploding gradients: Gradients can also grow exponentially, causing instability.

Practical solutions (not mathematically rigorous, but effective):

• Residual connections: Direct shortcuts for gradient flow
• Gradient clipping: Cap gradient magnitudes
• Normalization: Batch/layer normalization for stability

These aren't theoretically pure, but they make models trainable in practice.

7. Automatic Differentiation (AutoGrad)

What makes DL libraries such as PyTorch special? Beyond its many features, the core breakthrough is automatic gradient calculation for tensors and layers, essential for backpropagation.

Gradient calculation isn't magic. By constructing computational graphs, we can calculate gradients scalably using the chain rule. I recommend checking out the following resources:

• https://videolectures.net/videos/deeplearning2017_johnson_automatic_differentiation
• https://dlsys.cs.washington.edu/pdf/lecture4.pdf

8. Gradients: Direction and Step Size

We've emphasized that derivatives tell us direction, but look closer at the equations - gradients also determine step size (aside from the learning rate). To be more precise, relative magnitude.

• In ideal functions: Large gradients (steep slopes) suggest big jumps toward critical points. Small gradients (flat regions) suggest cautious, smaller steps. This adaptive behavior works beautifully.
• In real loss landscapes: This can backfire spectacularly. Large gradients might cause overshooting, while the landscape's irregular nature makes fixed learning rates problematic.

Modern solutions:

• Adaptive learning rates: Respond to gradient magnitudes
• Learning rate schedules: Large rates early in training, smaller rates later
• Advanced optimizers: Adam, AdaGrad, RMSprop, etc...

These seem complicated. But, all these sophisticated techniques are still Taylor expansion variants at their core! So cool!