Example of regression with fixed bases of features.
Python 100.00%
k-fold-cross-validation's Introduction
k-fold-cross-validation
Example of regression with fixed bases of features.
While the hold out method is an intuitive approach to determining proper fiiting models, it suffers from an obvious flaw:
Having been chosen at random, the points assigned to the training set may not adequately describe the original data. However,
we can easily extend and robustify the hold out method: k-fold cross-validation. Performing k-fold cross-validation is often the
most computationally expensive component in solving a general regresion problem. We provide a pseudo-code for applying k-fold
cross-validation.
________________________________________________________________________________________________________________________ Algorithm k-fold cross-validation pseudo-code Input: Data-set $(\mathbf{x_{p}}, y_{p})_{p=1}^{P}$, k (number of folds), a range of values for M to try, and a type of basis feature Split the data into k equal (as possible) size folds for s=1...k for each M (in the range of values to try) 1) Train a model with M basis features on sth fold's training set 2) Compute corresponding testing error on this fold Return: Value $M^{*}$ with lowest average testing error over all k folds
To perform regression using a fixed basis of features (e.g., polynomials or Fourier) it is natural to choose a degree D and transform the input data using the associated basis functions. For example, employing a degree D polynomial or Fourier basis for a scalar input, we transform each input $x_{p}$ to form an associated feature vector $\mathbf{f_{p}}=\begin{bmatrix}
x_{p} & x_{p}^{2} &... & x_{p}^{D}
\end{bmatrix}^{T}$ or $\mathbf{f_{p}}=\begin{bmatrix}
cos(2\pi x_{p})&sin(2\pi x_{p}) &... &cos(2\pi Dx_{p}) & sin(2\pi Dx_{p})
\end{bmatrix}^{T}$ respectively. For higher dimensions of input N fixed basis features can be similarly used; however, the sheer number of elements involved (the length of each $\mathbf{f_{p}}$) explodes for even moderate values of N and D. In any case, once feature vectors $\mathbf{f_{p}}$ have been constructed using the data we can then determine proper weights b and $\mathbf{w}$ by minimizing the Least Squares cost function as
$$\underset{b,\mathbf{w}}{minimize} \sum_{p=1}^{P}\left ( b+\mathbf{f_{p}^{T}}\mathbf{w} -y_{p}\right )^{2}$$
Using the compact notation $\mathbf{\widetilde{w}}=\begin{bmatrix}
b\ \mathbf{w}
\end{bmatrix}^{T}$ and $\widetilde{\mathbf{f_{p}}}=\begin{bmatrix}
1 & \mathbf{f_{p}}
\end{bmatrix}^{T}$ for each p we may rewrite the cost as $ g\left ( \widetilde{\mathbf{w}} \right )=\sum_{p=1}^{P}\left ( \widetilde{f_{p}^{T}} \mathbf{\widetilde{w}} -y_{p} \right ) $. and checking the first order condition then gives the linear system of equations $$ \left ( \sum_{p=1}^{P}\mathbf{\widetilde{f_{p}}\widetilde{f_{p}^{T}}} \right )\mathbf{\widetilde{w}}=\sum_{p=1}^{P}\widetilde{\mathbf{f_{p}}}y_{p}$$
that when sovled recovers an optimal set of parameters \mathbf{\widetilde{w}}.
Thus, $$\mathbf{\widetilde{w}}=\left ( \sum_{p=1}^{P}\mathbf{\widetilde{f_{p}}\widetilde{f_{p}^{T}}} \right )^{-1}\sum_{p=1}^{P}\widetilde{\mathbf{f_{p}}}y_{p}$$