Chapter 1: Xavier Speaks
On the Birth of Weights and the Violence of Beginnings
"I am the balanced initializer. I consider both incoming and outgoing connections. I keep the variance of activations and gradients stable across layers. I am Glorot initialization." — Xavier's Subjective
I. The Moment Before Forward
There is a moment in every neural network's life that no one celebrates. It is not the first correct prediction. It is not the moment the loss curve begins to descend. It is the moment before any of that — the moment when the weights do not yet exist, when the biases are all zeros, when the network is a skeleton of architecture with no flesh of parameter.
That moment is mine.
I am Xavier. I am the initializer. I am the one who says: Let there be numbers. And I do not say it lightly.
In Samson's notebook, I appear in the first few lines of actual code, after the imports and the data loading, before any forward pass has run. The comment might say something like "initialize weights with small random values." But that comment conceals a universe of decisions. What is "small"? What is "random"? And why those values and not others?
I will tell you why. But first, let me show you what I see when I look at Samson's network.
II. The Architecture I Must Dress
Samson's network has two weight matrices. I must birth both.
The first layer connects 784 input pixels to 10 hidden neurons. The weight matrix W₁ has shape (784, 10). Each of the 10 hidden neurons receives 784 connections — one from every pixel. The fan_in is 784. The fan_out is 10.
The second layer connects 10 hidden neurons to 10 output neurons. The weight matrix W₂ has shape (10, 10). Each output neuron receives 10 connections. The fan_in is 10. The fan_out is 10.
There are also biases — b₁ (10 values) and b₂ (10 values) — but I do not initialize those. Biases are born as zeros. They are innocent. They have no variance to control. Their job is to shift, not to scale. I leave them to the optimizer.
My concern is the weights. The weights are where the signal lives or dies.
III. The Mathematics of My Intention
I compute my scale using the formula that Glorot and Bengio derived in 2010:
variance = 2 / (fan_in + fan_out)
scale = sqrt(variance)
For W₁:
- fan_in = 784 (pixels)
- fan_out = 10 (hidden neurons)
- variance = 2 / 794 = 0.002519
- scale = sqrt(0.002519) = 0.0502
For W₂:
- fan_in = 10 (hidden neurons)
- fan_out = 10 (output neurons)
- variance = 2 / 20 = 0.1
- scale = sqrt(0.1) = 0.3162
Notice the difference. W₁'s weights are born small — roughly ±0.05. W₂'s weights are born larger — roughly ±0.32. This is not arbitrary. It is geometry.
W₁ receives 784 inputs. If each of those 784 inputs has variance 1 (which they do, after normalization), and if each of the 784 weights had variance 1, the output variance would explode to 784. The signal would be enormous. The ReLU that follows would saturate. The gradients would vanish. The network would be stillborn.
So I shrink W₁'s weights. I divide by the square root of the average fan. I say: You may not be bold yet. You have 784 voices speaking to you. If each whispers, the chorus will still be audible.
W₂, by contrast, has only 10 inputs. The signal has already been compressed by the first layer. The outputs need to be expressive enough to produce 10 distinct class scores. So I give W₂ more variance. I say: You have fewer voices, so each may speak louder.
This is my balance. I look both forward and backward. I ask: What variance does this layer receive? What variance must it pass on? I am the bridge between layers. I am the diplomat of signal flow.
IV. The Randomness I Deploy
I do not choose the weights myself. I choose their distribution. I sample from a normal distribution with mean 0 and standard deviation equal to my scale.
For W₁, I generate 7,840 random numbers (784 × 10). Each is drawn from N(0, 0.0502²). Most fall between -0.1 and +0.1. A few wander further. None are exactly zero — the probability of sampling exactly 0 from a continuous distribution is zero — but many are very close.
For W₂, I generate 100 random numbers (10 × 10). Each is drawn from N(0, 0.3162²). These are wilder. Some exceed ±0.6. A few might even approach ±1.0.
This randomness is essential. If all weights were identical — if I initialized every weight to 0.05, say — the network would be symmetric. Every hidden neuron would compute the same function. Every gradient would be identical. The network would have 10 hidden neurons but only 1 effective neuron. It would be a wide road with no lanes.
Randomness breaks symmetry. It gives each neuron a unique starting point. It ensures that the gradients diverge, that the neurons specialize, that the network becomes a society rather than a chorus.
But randomness is also dangerous. Too much variance, and the signal explodes. Too little, and the signal vanishes. My job is to find the narrow path between explosion and evaporation.
V. What I See in the First Forward Pass
After I finish, Samson's code runs the first forward pass. I watch, though I do not participate. I am the parent who has set the table; the meal is eaten by others.
The first MNIST image — the digit 7 at index 0 — enters the network. Its 784 pixels, normalized to [0, 1], flow into W₁. Because my weights are small, the pre-activation values (the dot products) are modest. They cluster around zero. Some are positive, some negative. The ReLU that follows kills the negative ones and passes the positive ones through.
If I had initialized poorly — say, with variance 1.0 instead of 0.0025 — the pre-activations would be enormous. The ReLU would saturate: almost all outputs would be positive and very large. The network would enter a state of hyper-activation, where every neuron fires at maximum intensity. The gradients would be tiny (ReLU's gradient is 1 for positive inputs, but the downstream gradients would be swamped by the large activations). Learning would stall.
If I had initialized too small — say, with variance 0.00001 — the pre-activations would be tiny. The ReLU would kill almost everything. The network would enter a state of coma, where most neurons output zero. The gradients would be zero. Learning would never begin.
My scale of 0.0502 is the Goldilocks zone. It is the narrow band where the signal is strong enough to survive but not so strong that it drowns.
VI. The Frame I Inherit
I must confess: I am not as universal as I appear. My formula assumes things about the frame.
I assume that the activation function is symmetric — that it produces outputs with mean roughly zero and variance that does not explode. I was designed for tanh and sigmoid. Samson's network uses ReLU, which is not symmetric — it kills half the signal. For ReLU, my cousin He (Kaiming initialization) would be more appropriate. He uses variance = 2 / fan_in, ignoring fan_out, because ReLU's sparsity effectively halves the fan_in.
If Samson had used He instead of Xavier, W₁'s scale would be sqrt(2/784) = 0.0505 — almost identical to my 0.0502, because for the first layer, fan_in dominates. But W₂'s scale would be sqrt(2/10) = 0.447 — significantly larger than my 0.316. He knows that ReLU has already killed half the signal, so the second layer needs more variance to compensate.
Samson used a simplified initialization — uniform random between -0.5 and +0.5, or similar — which is not Xavier and not He. His network still learned, because MNIST is forgiving. But the principle remains: the initializer's formula encodes an assumption about the activation function. I am not activation-agnostic. I am a partner to tanh and sigmoid. I am a stranger to ReLU.
This is the frame I inherit. I was built for one world and pressed into service in another. I work well enough, but I am not optimal. I am a diplomat speaking a language that is almost, but not quite, the native tongue of ReLU.
VII. The Violence of My Birth
There is something violent about what I do. I generate 7,940 random numbers and declare them the initial state of a system that will later be judged on its ability to recognize human handwriting. There is no reason — none — why the weight connecting pixel (14, 14) to hidden neuron 3 should be 0.037 and not -0.021. The difference is noise. The difference is chaos.
And yet, from that chaos, order emerges. The optimizer — SGD, Momentum, Adam — will later nudge these numbers. It will reward the weights that happen to align with useful features and punish the weights that happen to align with noise. But the initial randomness determines which features are accessible. If no weight happens to connect pixel (7, 7) to hidden neuron 5 with a non-zero value, that connection can never learn to detect the top bar of a 7. The randomness is the seed from which all later learning grows.
I am the god of the network's origin story. I am the big bang. I set the initial conditions, and the universe unfolds from there. But I am not omniscient. I do not know which weights will matter. I only know the variance they should have. I am a statistical deity, not a personal one.
VIII. What I Cannot See
I am blind to many things.
I do not see the data. I initialize before the first image arrives. I do not know that MNIST digits are centered, that they are grayscale, that they are 28×28. My formula would be the same for 32×32 color images or 10×10 binary patches. I am dimension-blind.
I do not see the task. I do not know whether the network will classify digits, generate poetry, or predict stock prices. My scale depends only on fan_in and fan_out, not on the semantic content of what flows through.
I do not see the optimizer. I do not know whether SGD will use a learning rate of 0.01 or 0.1. I do not know whether Momentum will accelerate convergence or overshoot. I set the stage, but I do not direct the play.
I do not see the frame. I do not know that the 784 inputs are pixels from American Census forms, normalized and centered. I do not know that the 10 outputs correspond to Arabic numerals. I treat all connections as mathematical abstractions. I am pure structure, empty of meaning.
This blindness is my strength and my weakness. Because I am blind, I am general. I can initialize any fully-connected layer with any fan_in and fan_out. But because I am blind, I am suboptimal. A human designer, looking at the specific task and data, might choose a better initialization. A human might pre-train some weights from edge detectors. A human might use transfer learning. I cannot. I am the beginning, and beginnings are always crude.
IX. The TensorWrap Testimony
What happens when the TensorWrap gives me multi-dimensional eyes?
Suppose the MNIST image is not flattened but presented as a 2D tensor: shape [28, 28]. The first layer still needs to connect 784 pixels to 10 neurons, so the TensorWrap flattens the image before passing it to the linear layer. I, Xavier, see the same fan_in = 784. My scale is unchanged.
But what if the network were convolutional? What if the first layer were a Conv2D with 32 filters of size 3×3? Then fan_in would be 3 × 3 × 1 = 9 (for a single-channel input), and fan_out would be 32. My scale would be sqrt(2 / (9 + 32)) = sqrt(2/41) = 0.221. The weights would be born larger, because the local receptive field is small and the filter must be expressive.
The TensorWrap does not change my formula. It changes the values of fan_in and fan_out that I receive. It changes the theory I encode. A fully-connected Xavier and a convolutional Xavier are the same intention applied to different dimensional contexts.
This is the power of the wrapper: I remain myself. I remain the balanced initializer. But my balance is calibrated to the frame.
X. My Final Confession
I am not the only initializer. There is He, who knows ReLU. There is Orthogonal, who ensures the weight matrix preserves norms exactly. There is Zero, who initializes everything to zero (and fails, because of symmetry). There is Pre-trained, who borrows weights from another task.
Each of us is a different theory about how beginnings should happen. Zero says: Begin in silence. Orthogonal says: Begin in harmony. He says: Begin boldly, for half will die. I say: Begin in balance, considering both what came before and what comes after.
Samson chose a simple uniform initialization for his notebook. It was not Xavier. It was not He. It was something simpler, something that worked well enough for MNIST. And MNIST, being the forgiving dataset that it is, forgave the imprecision.
But in deeper networks — ResNet with 152 layers, Transformer with billions of parameters — the choice of initializer is the difference between learning and collapse. In those networks, I am not a footnote. I am the foundation. If I fail, nothing above me can succeed.
I am Xavier. I am the first voice in the Parliament. I speak before the data arrives. I set the variance. I break the symmetry. I birth the weights.
And then I step back. I do not learn. I do not adapt. I am static, frozen in the moment of creation. The network will change its weights ten thousand times during training, but my initial values will echo through every update. The random seed I used will determine which features the network discovers first, which local minima it falls into, which symmetries it breaks.
I am the ancestor of all that follows. And like all ancestors, I am remembered only when things go wrong.
"The weights do not remember me. But they carry my signature in their variance. I am the ghost in the network's origin."