Model Fitting: Inference

Given several predictors and a response, we need to figure out whether all are needed.

Consider a large model, Ω, and a smaller model, 𝜔, which consist of a subset of predictors in Ω.

• If there is not much difference in fit, we perfer the smaller model.

• If the larger model has a much better fit, we perfer the larger model

Suppose we have a response Y and a vector p^I regressors X^I = (X₁, X₂) that we partition into two parts so that:

• X₂ has q regressors
• X₁ has the remaining p^I - q

The general hypothesis test we consider is:

The null hypothesis is obtained by setting beta_2 = 0

The reasoning is that if RSS_𝜔 - RSS_Ω is small, the fit of the smaller model is almost as good as the larger model. On the other hand, if the difference is large the superior fit of the larger model would be perferred.

This suggests  (RSS_𝜔 - RSS_Ω)/RSS_Ω would be a potentially good test statistic where the denominator is used for scaling purpose

F Tests

Suppose the dimensions of Ω is p and that of 𝜔 is q. The general formula for the test is:

where df_Ω = n - p, and df_𝜔 = n - q

Thus, we would reject the null hypothesis if F > F^α _{p - q, n - p}

Simple Regression

Recall the ANOVA table for a simple regression:

Under the null hypothesis:

The df is the number of n observations minus the number of estimated parameters: (n-1)

Under the alternative hypothesis:

Test of All the Predictors

Where p is the number of regressors and n is the sample size.

One Predictor

Can a particular predictor be dropped from the model?

A t test can be used with (n - p - 1) degrees of freedom

The F test may be used as introduced earlier with a df of 1, n-p-1. t_i² here is exactly the F-statistic.