Overfitting basics
Overfitting is a statistical phenomenon that occurs when you train on or make decisions on a dataset. The model or system that results from that data set learns patterns that are specific to that data set and do not extrapolate.
This is most severe on a training set, but can just as easily occur on a separate "validation" set that is used to make decisions about a model or system.
The solution to overfitting is measuring the final outcome on a dataset that the model has never seen before - e.g., a test set. A holdout test set should be used sparingly, only to evaluate the final result. If the test set is used only once, the measurement is reliable.
The training data splits
The standard machine learning training pattern is to create a training set, a validation set, and a test set.
- Training set: The model learns directly from these data points and has visibility to each.
- Validation set: The model never sees these data points, but the dataset is used to make decisions about the parameters in the model or which models to pick out of a group of models.
- Test set: This data set is used once and only once.
Avoiding Data Mining
When a validation set is used sparingly, the danger of overfitting the validation set is low. A trusted approach is to make some decision that is guided by theory and test it on the validation set. For example, you take a theory about the stock market: “I believe that companies that have production on U.S. soil will outperform in the U.S. when tariff risks are impending.” You test that theory out with 5 or 10 theories based in logic. This is reasonable.
However, data mining occurs when you test many random hypotheses without any theoretical background. Let's say you run 100 experiments and you say you're only going to accept experiments that are statistically significant. That significance actually comes with an assumption. In fact, it says that (with an alpha value of 0.05), 5% of statistically significant results will be due to chance alone. If you run 100 random experiments, you will actually end up with five statistically significant results on average, due to chance alone.
Adaptive reuse
A more advanced topic is called adaptive reuse. Adaptive reuse occurs when you make iterative improvements against a dataset. Greedy optimizations iteratively improves a system against a validation set, which is classified as adaptive reuse, and the expected over-fitting scales based on the formula:
Error ≤ C * √(k / n)
C: Constant.
k: Number of validation set runs
n: Validation set sample size.
The constant C will depend on the characteristics of your overfitting. How much information are you actually extracting from the validation set? Values can be less than one; they can be greater than one. It's not very easy to measure this.
Here's an example:
Suppose your validation set has 150 rows (n), and your greedy optimization runs for 30 iterations (k).
error ≤ C * √(15 / 150)
error, upper-bound = C * 31%
Versus only 5 sequential validation set tests:
error ≤ C * √(5 / 150)
error, upper-bound = C * 18%
Non-adaptive reuse
Meanwhile, when the changes are non-adaptive, (for example, running non-iterative experiments) the formula differs, and the error is much more easily reduced by your sample size.
error ≤ C * √(log(k) / n)
error ≤ C * √(log(5) / 150)
error, upper-bound = C * 5%
This shows that when the validation set is reused in a non-iterative way (for example, if the experiments are all run on the same initial base program), the overfitting error is much smaller.
In all cases, the constant is fully unknown (it depends on the use case and can't be measured). Therefore, the point is not to estimate the amount of overfitting, but to view how changes to the dataset size impact the amount of overfitting. When planning an optimization run, it is valuable to keep the number of iterations within a reasonable range, and to use more data whenever possible.
Takeaways
- Sequential iterations (aka. adaptive reuse) result in more overfitting than non-sequential iterations.
- An evolutionary approach provides multiple non-sequential branches, reducing the number of sequential iterations. The total k for adaptive reuse would be mostly based on the size of the chosen trajectory branch rather than the total number of iterations.
- CodeEvolver limits overfitting and sequential iterations by taking an evolutionary approach (less adaptive reuse) and by publishing final holdout test set results.
Generally speaking, we want to keep our trajectories relatively short, informed by theory, with at least 100 rows in the validation set. For the training set, 100 samples in the training set would provide a standard 50/50 split. For the test set, you might not need a holdout set to benefit, but a test set will measure overfitting. The test set can be any size. The larger the size, the more precise you can get. Thus I’d start with 100/100 if you have limited data, and 100/150/100 (and more) for clients that can make the investment.