Artificial Neural Networks: Mathematical Perceptrons, Backpropagation & Deep Architectures

Artificial Neural Networks (ANNs) form the computational backbone of modern Deep Learning. By composing stacked non-linear transformations, neural networks approximate arbitrarily complex functions across continuous manifolds (The Universal Approximation Theorem).

1. The Artificial Neuron (Perceptron) Formulation

Inspired loosely by biological dendritic inputs, synaptic cleft integration, and axon hillock action potentials, the formal mathematical model of an artificial neuron computes a linear combination of inputs parameterized by weights W and bias b, passed through a non-linear activation function σ:

z = ∑_(i=1)^d w_i x_i + b = W^T x + b | a = σ(z)

Inputs (x)

Feature attributes or activations from the preceding layer.

Weights (W)

Learnable synaptic connection strengths updated during backprop.

Bias (b)

Affine scalar shift permitting activation threshold adjustment.

Activation (σ)

Introduces non-linearity; prevents layer collapse into a single matrix.

2. The Mathematical Mechanics of Backpropagation

Backpropagation is the application of the multivariable calculus chain rule to calculate the partial derivative of the scalar loss L with respect to every weight parameter in the network graph:

∂L / ∂w_ij^(l) = δ_i^(l) · a_j^(l-1) | where δ_i^(l) = (∂L / ∂z_i^(l)) = ∑_k δ_k^(l+1) w_ki^(l+1) · σ'(z_i^(l))

Step 1: Forward Propagation Pass

Inputs x flow forward through the layers. Each layer computes z^(l) = W^(l) a^(l-1) + b^(l)and applies a^(l) = σ(z^(l)). Intermediate activations are cached in memory for backward pass calculation.

Step 2: Backward Error Propagation

The loss derivative ∂L/∂a^(L) is evaluated at the output. Gradients propagate backwards layer-by-layer. Optimizers (SGD, AdamW) use these gradients to adjust parameter weights in the direction that minimizes loss.

3. Comparative Analysis of Activation Functions

Function	Formula	Range	Advantages & Primary Drawbacks
Sigmoid	1 / (1 + e^(-z))	(0, 1)	Outputs valid probability; suffers from severe vanishing gradients when \|z\| is large.
Tanh	(e^z - e^(-z)) / (e^z + e^(-z))	(-1, 1)	Zero-centered output accelerates gradient descent; still suffers from vanishing gradients.
ReLU	max(0, z)	[0, ∞)	Computationally cheap, mitigates vanishing gradients; vulnerable to "Dying ReLU" when z < 0.
GELU	z · Φ(z) ≈ 0.5z(1 + tanh(√(2/π)(z + 0.044715z^3)))	(-0.17, ∞)	Smooth probabilistic non-linearity; default standard in BERT, GPT, and modern Transformers.