Model Fitting: Inference

Given several predictors and a response, we need to figure out whether all are needed.

Consider a large model, Ω, and a smaller model, 𝜔, which consist of a subset of predictors in Ω.

• If there is not much difference in fit, we perfer the smaller model.

• If the larger model has a much better fit, we perfer the larger model

Suppose we have a response Y and a vector p^I regressors X^I = (X₁, X₂) that we partition into two parts so that:

• X₂ has q regressors
• X₁ has the remaining p^I - q

The general hypothesis test we consider is:

The null hypothesis is obtained by setting beta_2 = 0

The reasoning is that if RSS_𝜔 - RSS_Ω is small, the fit of the smaller model is almost as good as the larger model. On the other hand, if the difference is large the superior fit of the larger model would be perferred.

This suggests  (RSS_𝜔 - RSS_Ω)/RSS_Ω would be a potentially good test statistic where the denominator is used for scaling purpose

F Tests

Suppose the dimensions of Ω is p and that of 𝜔 is q. The general formula for the test is:

where df_Ω = n - p, and df_𝜔 = n - q

Thus, we would reject the null hypothesis if F > F^α _{p - q, n - p}