Very frequent
0/14
Model Evaluation
Syllabus domain seven: the confusion matrix, cross-validation, and building a validation split you can trust in 90 minutes.
Syllabus domain seven: the confusion matrix, cross-validation, and building a validation split you can trust in 90 minutes.
8 multiple choice at 2 marks and 4 fill-in-the-blank at 5, marked exactly as Round 1 marks them. Each answer is explained as soon as you check it. The clock is shown, not enforced.
Round 2 is scored on model performance, and Round 1 asks you to compute these quantities by hand from a confusion matrix. Both are reliable marks.
Given a matrix, be able to produce all four quickly. The most common slip is confusing the precision and recall denominators: precision divides by everything you predicted positive, recall by everything that was positive.
If Round 2's dataset is imbalanced, say so in your notebook and report an appropriate metric. Demonstrating that you noticed is part of what code quality means here.
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipeline, X, y, cv=cv, scoring="f1_macro")
print(scores.mean(), scores.std())
Stratified k-fold is the default for classification, and mandatory when classes are imbalanced. Use GroupKFold when observations cluster, and TimeSeriesSplit when the data is temporal - a random split on either leaks.
Report the standard deviation as well as the mean. A model scoring is not clearly better than one scoring , and noticing that is worth more than a hundredth of accuracy.
Full 5-fold cross-validation on every candidate is not affordable. A workable compromise:
If training score is far above validation score, you are overfitting. The fixes, in order of cost: reduce model complexity, add regularisation, reduce features, get more data. In ninety minutes the first two are the realistic options.
Know the precision and recall denominators cold; they are the common slip. Accuracy is the wrong headline metric under imbalance - name a better one.
Stratify classification folds, and group or order them when the data demands. Report both the mean and the variance across folds.
A model flags 10 rows as positive; 6 of those flags are correct. What is precision?
Precision , so precision is .
Why stratify a classification split?
Each fold (or the holdout) keeps the same positive rate. A rare class cannot vanish from a fold by chance.
Select an answer