Validation set's split structure selects the model

The validation set says: “I’ll tell you which candidate scores best by my criterion”, or answers the question: “Which model is the best?”. Random val? The best interpolator wins. Temporal val? The best forecaster wins. Unit-grouped val? The best new-unit generalizer wins.

The validation set is used in the training loop to evaluate batches of data. Based on those evaluations (specifically the target metric), the best model is chosen, so validation set is actively used during training and has a real impact on the model choice. That chosen model is later evaluated using the test set.

Rests on

Test set's split structure determines the claim about the model info on what role the test set has.
Random splitting temporal data makes the target interpolation not forecasting a consequence of not thinking it through.
The split type has to preserve the structure you're generalizing along a general rule on splitting datasets
Validation and Test sets must have identical split structure to support the same claim the dependency on the validation dataset.

Cited by

Test set's split structure determines the claim about the model info on what role the validation set has.
Validation and Test sets must have identical split structure to support the same claim the role of the validation set in model training.