A replication knob must be both an arbitrary choice and unrelated to the target variable

Arbitrary: The choice must not be derived from any prior knowledge, or dependency on the system (i.e. model architecture, data, etc). If that condition is not satisfied, we’re not measuring noise across runs, but how the model responds to educated decisions.

Unrelated to target: The choice must be independent. You could re-use a random seed 42 and see better results as that particular setup (downstream subsampling, folding etc.) happens to produce a biased result. In this case - by luck. If you select initial NN weights that are informed in some way on the target (by chance or by design), that would also violate the rule and therefore, you’d end up measuring something more than just noise across runs.