Motivation
An architecture restricts how parameters connect computations. A convolution uses the same local weights at many spatial positions. A recurrent model uses the same transition parameters at several times. These restrictions can greatly reduce parameter count, but they also make a mathematical commitment: a pattern or update rule is reused rather than fitted independently at every location.
This is more precise than calling one architecture suitable for images and another suitable for sequences. We can ask which transformations a filter respects, what information a recurrent state retains, and how every reuse contributes to a gradient. A concrete application should exhibit its input, operator and output before a claim about its usefulness is made.
The finite circular filters and scalar recurrent models used here establish exact operator properties, not empirical superiority or a complete modern architecture survey.
Definitions
Definition 31.1.1
(Shared correlation operator on a circular grid)
Take a one-dimensional signal with positions indexed modulo . A shared correlation operator is , where is a finite set of offsets and the same kernel and bias are used at every position. Modular indexing specifies circular boundaries. For multiple channels, the corresponding operator is
Parameters depend on output channel, input channel and offset, but not position. There are parameters, instead of one independent local kernel at each of positions. The term convolution is often used for this unflipped correlation in neural software; the indexing convention here fixes the signs unambiguously.
Definition 31.1.2
(Translation, equivariance and invariance)
Define translation by . An operator is translation equivariant when . A scalar statistic is translation invariant when its value is unchanged. Equivariance preserves where a feature moves; invariance removes location from the output. Circular boundaries are part of the exact theorem. Zero-padded finite arrays can fail the same identity near their boundaries.
Definition 31.1.3
(Finite scalar recurrent model)
For a scalar recurrent model, fix initial state independently of the trainable parameters and write
The input sequence is finite and fixed, and all functions are differentiable at the evaluated arguments. The shared parameters are reused at every time. Loss terms may be zero at times with no supervision. Unrolling this finite recurrence gives an acyclic computational graph; it is not differentiation through an unresolved algebraic cycle.
Theorem/Proposition
Theorem 31.1.4
(Equivariance of shared stride-one operators)
On the circular grid, a shared stride-one correlation with spatially constant bias commutes with every translation. Pointwise activation functions applied identically at every position also commute with translation; compositions preserve equivariance. Averaging all spatial positions of an equivariant channel gives a translation-invariant output. Subsampling preserves a corresponding equivariance only for shifts compatible with its sampling lattice.
Theorem 31.1.5
(Gradients for shared filters and a finite recurrence)
For one filter channel with incoming adjoints ,
For the recurrent model, put and compute backwards
Then the gradients for a,b,c,d are respectively , , , and . All sums use the same old parameter values.
Concept lensAlgebraic
A shared correlation is a circulant matrix
Write the input as a vector in . The shared correlation is multiplication by the matrix with (indices modulo ) and all other entries zero, plus the constant vector . Its entries depend only on modulo , which is exactly the statement that the same kernel is used at every position. Conversely, a linear map commuting with every translation satisfies for all , so its entries depend only on and it is a shared correlation with offsets taken over all residues. For the filter on , offsets , the rows of are , and , so .
Proof sketch or proof idea
Proof
(Translation commutes with a shared filter)
For translation, the filtered shifted signal at i is , exactly the original filtered signal at . A pointwise activation acts on equal translated values and preserves the equality. Apply this argument successively through a composition. Spatial averaging is invariant because translation permutes the finite set of indices. The proof identifies why sharing and uniform bias matter: position-dependent weights or biases need not survive the index change.
If n is divisible by a stride q and subsampling is , then on the coarse circular grid. A one-pixel shift need not correspond to any coarse shift. Thus downsampling can preserve useful structure without retaining the full original translation group. Statements of architectural symmetry should name the actual transformation set.
Proof
(Collecting the differentials of a shared filter)
For filter differentiation, substitute into . Collect coefficients of each shared kernel differential to obtain the first formula. Reindex to collect each input differential and obtain the second. The bias coefficient sums over all uses as well. For several channels the same argument includes the appropriate channel sums; parameter sharing never means averaging contributions unless the loss itself has an averaging factor.
Proof
(Backward induction over time)
For recurrence, the loss depends on directly through and indirectly through . The corresponding derivative with respect to is . Multiplying by gives . Backward induction from the final time proves every adjoint. Collecting the local coefficients of each shared parameter over time gives the four gradient sums.
Proof X-ray
One parameter has many spatial or temporal uses
Each use contributes a local differential. Since all those differentials represent the same parameter perturbation, their coefficients must be added. Treating the uses as independent weights silently enlarges the architecture; using only the last time step omits earlier routes. A finite unrolled graph makes this accumulation explicit and justifies backpropagation through time.
The recurrent derivative also assumes the input sequence is held fixed. In a closed-loop predictor, an input may instead contain the previous prediction. For example, with identity activation and feedback input , the transition becomes . Its state sensitivity is governed by , rather than a alone. Ignoring that feedback path differentiates a different graph. Feeding observed inputs during training and generated inputs during deployment can therefore change both the forward trajectories and their sensitivities. The finite fixed-input theorem remains correct; it must simply not be applied after silently changing what depends on the parameters.
Sensitivity over a long sequence has another exact expression. Holding inputs fixed, in this scalar model. If along the trajectory, its magnitude is at most . When , this particular long-range sensitivity vanishes exponentially. For identity activation it is exactly , so values can instead amplify it. These statements concern one sensitivity, not every full loss gradient: sums, output factors and cancellations also matter.
Sharing can be checked with a zero input. A zero signal through a shared kernel and a constant bias gives the same output at every position, so translating it changes nothing. If each position had its own bias, the zero input would still be unchanged by every translation, but the output would be a list of unequal biases; translating that output generally changes it, so the equivariance identity fails at once. The bias is therefore part of the architectural constraint, not only the kernel weights.
The boundary policy of a finite image is also part of the operator. A circular model joins opposite edges, which suits the index shift used in the theorem; a zero-padded model treats the outside as zero, and positions touching the boundary see different data, so equivariance is not unconditional. If an application uses only interior positions far from the boundary, a local translation relation can be proved separately for those positions, but that local conclusion does not extend to unconditional equivariance of the whole array. Choose the operator by the meaning of the boundary in the task before choosing a convenient code form.
A recurrent application must also state when a sequence ends and when the state resets. If two independent samples share the final state of the previous one, the second output depends on the first sample, and the model is no longer started from the fixed initial state of the theorem. When a continuous data stream is processed deliberately, keeping the state can be reasonable, but the training graph and the evaluation split must acknowledge that dependence. State memory is not a free or unconditionally correct source of information.
This unit holds the initial state fixed, so it is not updated. If it were trainable, one could still treat it as a source in the unrolled graph, and the backward pass would have to continue to the first step and return the influence to that initial state; that is an additional parameter. List which quantities are fixed, which are shared and which are trainable before asking whether a gradient is complete.
Worked examples
Worked example 31.1.6
(Training one shared filter at all positions)
Take n equal to three, , offsets 0 and 1, kernel and zero fixed bias. The responses are . With zero targets and , the loss is 3 and incoming adjoints equal z. Kernel gradients are and . A simultaneous step gives , responses and loss . All positions contributed to each learned coefficient; fitting only the first response would produce a different update.
Worked example 31.1.7
(A recurrent state computes a running sum)
Use identity activation with . Then induction gives , so the output is the running sum. This is a complete mathematical sequence application with a fixed-size state. The state preserves the sum, not the whole sequence: inputs and have the same final state. If the task asks which number appeared first, this particular state cannot answer it. Architecture usefulness is relative to the required statistic.
Margin Note
Draw the unrolled graph first
Before differentiating a shared parameter, list every place it is used. Here appears at both times, although at it multiplies . The gradient is the sum over that list.
Worked example 31.1.8
(An exact recurrent training step)
Use identity activation, , inputs , parameters and final-only loss . Forward values are and loss . Backwards, . Gradients for are . Step gives . Recomputing gives , so loss is . This uses the old shared a in every backward recurrence, followed by one simultaneous update.
Common mistakes
Counterexample mode
Subsampling can destroy one-pixel equivariance
On four positions, let x have a single one at index zero. Keeping only even positions gives . Shift x by one pixel: its one moves to index one, and even-position subsampling now gives . No translation of equals the zero vector. A stride-two representation therefore cannot inherit arbitrary one-pixel equivariance from a preceding shared filter.
Common mistake
(Reading equivariance or a global average as an invariant answer)
A global average may help when location is irrelevant, but it removes information needed for localization. Conversely, equivariance alone does not make a classification output invariant. The task objective should determine which information is retained: locating a defect and merely reporting that one exists are different tasks.
Common mistake
(Presenting a truncated or clipped gradient as the exact one)
Truncating recurrent backpropagation omits routes crossing the truncation boundary. It can reduce computation, but generally changes the gradient of the stated full-sequence loss. Detaching a state is a deliberate derivative convention and should not be presented as exact full backpropagation. Clipping a computed gradient also changes the update, although it can be specified as a separate optimization rule.
Common mistake
(Confusing parameter count with gradient size)
Reusing a two-entry filter at a thousand positions still gives two filter parameters, but its loss gradient sums a thousand local contributions. If the objective is the mean per position rather than their sum, that normalization divides the gradient as well. Comparing training runs without recording this choice can make identical architectures appear to need inexplicably different step sizes. Sharing is a statement about which variables are equal, while averaging is a statement about the objective.
Finally, a bounded hidden state does not guarantee a nonvanishing derivative. Tanh outputs stay between minus one and one while its derivative becomes small in saturated regions. Nor does weight sharing prove that real data obey translation or time-homogeneous structure. These are useful modeling restrictions whose adequacy needs task-specific evidence. Operators with the same name need not be the same mathematical function, so the indexing, boundary, stride and place of sharing should be recorded with an implementation.
Summary
Shared operators impose exact reuse and, under stated boundary and sampling assumptions, exact symmetries. Differentiation adds contributions from every reuse. A recurrent computation becomes a finite graph when unrolled, yielding explicit backward sensitivities and gradient sums. Small complete applications reveal both what the state computes and what information it loses.
Exercises
Exercise 31.1.9
(For the filter example, compute the input gradient at all three positions using the theorem. Verify its sum is zero and explain the connection to adding a constant to every input.)
Core
Solution(3)
- 1
Compute the input derivatives
With kernel on offsets and adjoints , the formula gives , namely under modular indexing.
- 2
Check the sum
Key step
The entries sum to .
Because: Circular differences telescope around the cycle.
- 3
Connect to a constant shift
Adding a common constant leaves every difference unchanged, so the loss has zero directional derivative in the all-ones direction. The signed entries agree with that invariance.
Exercise 31.1.10
(Prove that applying the same ReLU at every spatial position preserves translation equivariance, even at inputs where ReLU is not differentiable. Does this proof require any gradient formula?)
FoundationProof
Solution(3)
- 1
Write the activated shifted signal
Let and let be any signal. Then .
- 2
Compare with the shifted activated signal
Key step
On the other side, , which is the same value at every index .
- 3
Read off the hypotheses
∎The two sides agree as function values, so commutes with translation. The argument used no derivative, and so it holds at the kink as well. A symmetry theorem and a gradient theorem can have different hypotheses; nondifferentiability at zero does not invalidate the former.
Exercise 31.1.11
(In the linear recurrent model with no input or bias, let and . Find the derivative with respect to . Compare it with the derivative with respect to the initial state and explain why these are different quantities.)
Core
Solution(3)
- 1
Differentiate with respect to the shared parameter
For a positive integer , , because is used times and each use contributes the product of the other factors.
- 2
Differentiate with respect to the initial state
Since with fixed, .
- 3
Compare
The first sums contributions from perturbing all uses of , while the second perturbs only the initial source. For the first derivative is for every , reading including at . The polynomial formula never requires dividing by ; at and it gives zero.
Exercise 31.1.12
(A running-sum model receives . Find all states. If only the final output is used to predict the sequence length, can this fixed state always suffice? Give two finite sequences that disprove such a claim.)
Core
Solution(3)
- 1
Compute the states
With and , the states are , , .
- 2
Find two sequences with the same final state
Key step
The sequences and both end at state but have lengths one and two.
- 3
Conclude
A predictor using only that final state must give both sequences the same answer, so it cannot always report the length. Additional state, such as a counter coordinate, changes the representation and can address this task.