Every data point is made of signal and noise
In other words, each data point represents some truth about the world, but also contains some randomness. The ratio of both depends on the data and the target variable we choose (truth about what?).
Rests on
- Extreme values regress toward the mean because they carry noise that doesn't repeat how it affects regression to the mean
- Averaging an ensemble's predictions preserves shared signal and cancels uncorrelated errors how noise is filtered out in ensemble methods
Cited by
- A permutation-invariant representation guarantees that the feature vector depends only on content and never on order a rule that's universal for any feature vector
- A single observation of extreme value can't reveal its own noise-to-signal ratio, but repeated observations can - by how far values pull back toward the mean
- Over-aggregation is irreversible, under-aggregation is recoverable what a data point is made of
- What's noise and what's signal in a data point depends on how the target variable is defined