An exploration of neural networks from mathematical foundations to modern deep learning pipelines.
Rather than treating deep learning libraries as black boxes, this repository focuses on the core principles underlying neural architectures: automatic differentiation, gradient backpropagation, statistical sequence modeling, dense representations, and the underlying hardware dataflow.
Every artificial neuron computes a linear combination of its inputs and weights, adds a bias term, and evaluates an activation function. In automatic differentiation systems, each mathematical operation is tracked as a node within a directed computational graph, enabling exact gradient evaluation with respect to every scalar parameter via the chain rule.
Stanford CS231n: Biological neuron model alongside its mathematical abstraction. Inputs are weighted, integrated with a bias, and transformed through non-linear activation.
In scalar-level engines, this mechanism calculates upstream and downstream derivatives across addition, multiplication, and non-linearities, providing the foundation for gradient-based optimization without manual derivative derivations.
A single neuron acts as a linear decision boundary. Stacking neurons into hidden layers allows networks to approximate complex, non-linear functions.
Stanford CS231n: Multi-layer architectures. Left: A two-layer neural network containing one hidden layer. Right: A three-layer network containing two hidden layers.
Regardless of depth, optimization follows a recurring three-phase cycle:
- Forward Pass: Input signals propagate through successive weight matrices and activations to produce predictions and evaluate scalar loss.
- Backward Pass: Error signals propagate backward through the computational graph via the chain rule, populating parameter gradients.
- Parameter Update: An optimizer nudges parameters opposite the direction of their gradients scaled by a learning rate.
Sequence prediction demonstrates how neural networks model discrete symbolic structures, such as natural text:
- Statistical Baseline (N-grams): Counting co-occurrence frequencies between adjacent tokens yields a direct probability distribution over the vocabulary. While fast, this method lacks generalization to unseen contexts and scales poorly with context size.
- Neural Language Modeling: Tokens are mapped to continuous, learned embedding spaces. A context window of historical token embeddings is projected through hidden layers and normalized via softmax to output probability distributions over potential succeeding characters.
Applying neural networks to high-dimensional data, such as image grids, illustrates the transition from scalar operations to batched tensor computations. Images are flattened into high-dimensional feature vectors, passed through dense hidden layers, and categorized into class probability distributions. Performance analysis via confusion matrices reveals characteristic failure modes and misclassification patterns across visually similar classes.
The end-to-end training process can be understood through systems and hardware analogies, mapping theoretical operations directly to hardware primitives:
| Phase | Operation | Systems Analogy | Function |
|---|---|---|---|
| Phase 1: Data Pipeline | Tokenization and Batching | DMA Controller | Streams sliding-window contiguous memory blocks into batched tensors |
| Phase 2: Embedding Lookup | Vector Mapping | BRAM / Look-Up Table | Maps discrete token identifiers to dense floating-point representations |
| Phase 3: Forward Pass | Matrix Multiplication | Systolic Array / GEMM | Executes batched linear transformations and computes prediction loss |
| Phase 4: Backward Pass | Backpropagation & Descent | Feedback Control Loop | Evaluates gradients through the graph and updates weight matrices |
- Python, NumPy, Matplotlib
- PyTorch
- TensorFlow, Keras
- Graphviz