AIMT-2 GATE DA 2024| ML Question for GATE | Wanshington Uni | Aug-23 Mid term test

Question

Suppose we are performing leave-one-out (LOO) validation and $10$-fold cross validation on a dataset of size $100, 000$ to pick between $4$ different values of a single hyperparameter. How many times greater is the number of models that need to be trained for LOO validation versus $10$-fold cross validation?
Answer:

Answer 1

LOO Validation:

• Trains a separate model for each data point, leaving that point out as the validation set.
• With a dataset of 100,000 points and 4 hyperparameter values, it trains 100,000 * 4 = 400,000 models.

10-Fold Cross Validation:

• Divides the dataset into 10 folds (subsets) of equal size.
• Trains 10 models, each using 9 folds for training and the remaining fold for validation.
• With 4 hyperparameter values, it trains 10 * 4 = 40 models.

Comparison:

• Number of models trained for LOO validation: 400,000
• Number of models trained for 10-fold cross validation: 40
• Ratio of models trained (LOO / 10-fold): 400,000 / 40 = 10,000

Therefore, LOO validation requires 10,000 times more models to be trained than 10-fold cross validation in this case.

Answer 2

In leave-one-out (LOO) validation, you train the model for each data point individually, leaving out one data point each time and using the remaining data for training. So, for a dataset of size 100000, you would need to train the model 100000 times.

In k-fold cross-validation, you divide the dataset into k folds and train the model k times, each time using a different fold as the validation set and the remaining folds for training. In this case, you mentioned 10-fold cross-validation, so you would need to train the model 10 times.

Now, to compare the number of models that need to be trained for LOO versus 10-fold cross-validation:

Number of models for LOO = 100000 Number of models for 10-fold cross-validation = 10

The ratio of the number of models for LOO to 10-fold cross-validation is:

10000010=1000010100000​=10000

So, the number of models that need to be trained for LOO validation is 10,000 times greater than the number needed for 10-fold cross-validation.