ریاضی | تمرین ۲

A model can fit a set of observations closely and still perform poorly when asked to predict circumstances that differ from those represented in the observations. One reason is that a flexible model can accommodate incidental irregularities as well as relationships likely to persist, without marking which part of its success depends on each. An exact match to the original record is therefore not an independent test of the assumptions used to produce that match.

Setting aside some observations before choosing a model creates an opportunity to examine predictions that were not directly used to guide the choice. This separation loses much of its force if the investigator repeatedly consults the reserved observations and adjusts the model until its performance on them becomes satisfactory. Although the reserved data may still be stored in a different file, they have become part of the process of selection rather than remaining a fresh challenge to its result.

Even a carefully preserved test has limits when the conditions of future use depart substantially from those under which all the observations were collected. A model developed from journeys on ordinary weekdays, for instance, may need further examination before it is used to predict travel during a major disruption. Evaluation thus involves both protecting a genuine distinction between selection and testing and asking how the tested conditions relate to the intended application. Recording which observations informed each revision can make this distinction easier to inspect, particularly when several people contribute to a long sequence of modeling decisions. That record does not guarantee future accuracy, but it makes the basis of a performance claim more open to critical examination.