Tensors: Memory Architecture, Strides & Operations

1. The Under-the-Hood Storage & Stride Architecture

A PyTorch tensor consists of two distinct components:

1. The Storage Buffer (torch.Storage): A 1D contiguous array of raw bytes allocated in host RAM or GPU VRAM containing the actual numerical values.
2. The Tensor Metadata View: Defines the shape, dtype, device, and strides that describe how to index into the 1D storage buffer to produce multidimensional coordinates.

import torch

# Create a 2D tensor of shape [3, 4]
x = torch.tensor([
    [10, 20, 30, 40],
    [50, 60, 70, 80],
    [90, 100, 110, 120]
], dtype=torch.float32)

print("Shape:", x.shape)        # torch.Size([3, 4])
print("Dimensions:", x.ndim)    # 2
print("Strides:", x.stride())   # (4, 1) -> Jump 4 elements for next row, 1 for next column!
print("Storage:", x.untyped_storage()) # Flat 1D array of 12 contiguous floats in RAM

2. Zero-Copy NumPy Bridging

When running on the CPU, PyTorch tensors and NumPy arrays share the exact same memory address. Modifying one mutates the other instantly without allocating new RAM or copying bytes:

import numpy as np
import torch

# 1. NumPy Array -> PyTorch Tensor (Zero-Copy)
np_arr = np.array([1.0, 2.0, 3.0, 4.0], dtype=np.float32)
torch_tensor = torch.from_numpy(np_arr)

# 2. Modify original NumPy array in place
np_arr[0] = 999.0

# 3. PyTorch tensor reflects the change immediately!
print("PyTorch Tensor reflects change:", torch_tensor[0].item()) # 999.0

# 4. PyTorch Tensor -> NumPy Array (Zero-Copy)
back_to_numpy = torch_tensor.numpy()

3. Essential Tensor Math & Matrix Multiplication

Deep learning relies on high-throughput linear algebra. Note the critical difference between element-wise multiplication and true matrix multiplication:

# Matrix Multiplication Example
A = torch.randn(32, 128)  # Batch of 32 vectors of dimension 128
W = torch.randn(128, 64)  # Weight matrix projecting 128 -> 64

# Compute linear projection
output = A @ W           # Shape: [32, 64]
print("Transformed Shape:", output.shape)

4. Device-Agnostic CUDA & Apple MPS Placement

In production machine learning, your Python script should automatically detect whether an NVIDIA GPU (CUDA), Apple Silicon GPU (MPS), or fallback CPU is available:

# Standard Device-Agnostic Boilerplate
def get_optimal_device() -> torch.device:
    if torch.cuda.is_available():
        return torch.device("cuda")
    elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
        return torch.device("mps")  # Apple Silicon Mac GPU
    else:
        return torch.device("cpu")

device = get_optimal_device()
print(f"Accelerating computation on: {device}")

# Move tensors to the target hardware device
X = torch.randn(2000, 2000, device=device)
Y = torch.randn(2000, 2000, device=device)

# Blazing-fast GPU matrix multiplication on Tensor Cores!
Z = torch.matmul(X, Y)
print(f"Completed computation on {Z.device}")