Study guide: Evaluate a Simple Classification Model
Use eight held-out cases with six correct predictions. Explain the accuracy and classify a reusable prediction whose label is not reusable. State one limitation.
Ideas to explain in your own words
- A feature is information supplied to a model. A training label identifies the outcome used in supervised learning examples. A prediction is the model’s output for a case. Using the target answer as an input can leak information and make a reported test misleading. Our dataset is fictional and fully supplied.
- Held-out cases are kept separate from fitting a model. Comparing predictions with their supplied labels counts correct and incorrect results. The result describes this test set; it does not guarantee performance on new populations or future data. Keeping the denominator visible prevents inflated summaries.
- A false positive predicts a positive outcome for a negative labeled case. A false negative predicts negative for a positive labeled case. Correct positive and correct negative are different from errors. The meanings of positive and negative must be defined before the labels are useful.
- Good performance on training examples may reflect fitting those examples rather than useful generalization. A held-out test set can provide different evidence, but only if it is not repeatedly used to tune choices. Report both set roles and their limitations. Our cases concern evidence quality, not a certified model.
Useful vocabulary
- feature
- Information supplied as model input.
- label
- A supplied outcome used for training or evaluation.
- held-out
- Kept apart from fitting and tuning.
Watch your reasoning
- The future answer should not be included as a feature when it is unavailable at prediction time.
- Do not discard wrong predictions from the denominator.
- A positive prediction is not proof that the labeled case is positive.
- Reusing test answers to tune a model makes that set part of model development.
A worked review example
Input includes recorded object size. Classify the field in the stated supervised model. The supported answer is Feature. Distinguish input information, known teaching outcome and model output before evaluating a case. The future answer should not be included as a feature when it is unavailable at prediction time. The supplied facts, rather than an assumed real-world detail, determine this paper case. Compare the recorded action or value with the stated rule before choosing an answer. Score on examples used to fit the model. Which evidence role is described? The supported answer is Training performance. Use an untouched evaluation set for the stated generalization check and disclose its scope. Reusing test answers to tune a model makes that set part of model development. The supplied facts, rather than an assumed real-world detail, determine this paper case. Compare the recorded action or value with the stated rule before choosing an answer.
Make a practice plan
- Make a four-row summary, one row for each lesson, with its computing idea and one worked decision.
- Explain one tempting mistake using the actual supplied example.
- Complete the new unit review before consulting its explanations.
- Try the transfer task on paper and compare your evidence against the model response.