# Ridge Regression¶

Coefficient estimates for multiple linear regression models rely on the independence of the model terms. When terms are correlated and the columns of the design matrix X have an approximate linear dependence, the matrix X’X becomes close to singular. As a result, the least-squares estimate becomes highly sensitive to random errors in the observed response Y, producing a large variance.

Ridge regression is one method to address these issues. In ridge regression, the matrix X’X is perturbed so as to make its determinant appreciably different from 0.

Ridge regression is a kind of Tikhonov regularization, which is the most commonly used method of regularization of ill-posed problems. Ridge regression shrinks the regression coefficients by imposing a penalty on their size. By allowing a small amount of bias in the estimates, more reasonable coefficients may often be obtained. Often, small amounts of bias lead to dramatic reductions in the variance of the estimated model coefficients.

Another interpretation of ridge regression is available through Bayesian estimation. In this setting the belief that weight should be small is coded into a prior distribution.

```
from miml import datasets
from miml.regression import RidgeRegression
x = array([[234.289, 235.6, 159.0, 107.608, 1947, 60.323],
[259.426, 232.5, 145.6, 108.632, 1948, 61.122],
[258.054, 368.2, 161.6, 109.773, 1949, 60.171],
[284.599, 335.1, 165.0, 110.929, 1950, 61.187],
[328.975, 209.9, 309.9, 112.075, 1951, 63.221],
[346.999, 193.2, 359.4, 113.270, 1952, 63.639],
[365.385, 187.0, 354.7, 115.094, 1953, 64.989],
[363.112, 357.8, 335.0, 116.219, 1954, 63.761],
[397.469, 290.4, 304.8, 117.388, 1955, 66.019],
[419.180, 282.2, 285.7, 118.734, 1956, 67.857],
[442.769, 293.6, 279.8, 120.445, 1957, 68.169],
[444.546, 468.1, 263.7, 121.950, 1958, 66.513],
[482.704, 381.3, 255.2, 123.366, 1959, 68.655],
[502.601, 393.1, 251.4, 125.368, 1960, 69.564],
[518.173, 480.6, 257.2, 127.852, 1961, 69.331],
[554.894, 400.7, 282.7, 130.081, 1962, 70.551]])
y = array([83.0, 88.5, 88.2, 89.5, 96.2, 98.1, 99.0, 100.0, 101.2,
104.6, 108.4, 110.8, 112.6, 114.2, 115.7, 116.9])
model = RidgeRegression(0.0057)
model.fit(x, y)
print(model.predict(x[:10,:]))
```

```
>>> run script...
array([83.71913397911655])
>>> model
Ridge Regression:
Residuals:
Min 1Q Median 3Q Max
-2.0691 -0.5736 0.2619 0.4844 1.6328
Coefficients:
Estimate Std. Error t value Pr(>|t|)
Intercept -247.2810 NA NA NA
Var 1 0.1789 7.8561 0.0228 0.9823
Var 2 0.0197 2.0319 0.0097 0.9925
Var 3 0.0066 0.8647 0.0076 0.9941
Var 4 -1.3433 4.2777 -0.3140 0.7607
Var 5 0.2216 9.7525 0.0227 0.9824
Var 6 -0.0575 3.7635 -0.0153 0.9881
---------------------------------------------------------------------
Significance codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.2361 on 9 degrees of freedom
Multiple R-squared: 0.9921, Adjusted R-squared: 0.9869
F-statistic: 189.0534 on 6 and 9 DF, p-value: 6.011e-09
```