Sliced inverse regression

Sliced inverse regression (SIR) is a tool for dimension reduction in the field of multivariate statistics.

In statistics, regression analysis is a popular way of studying the relationship between a response variable y and its explanatory variable ${\displaystyle {\underline {x}}}$, which is a p-dimensional vector. There are several approaches which come under the term of regression. For example parametric methods include multiple linear regression; non-parametric techniques include local smoothing.

With high-dimensional data (as p grows), the number of observations needed to use local smoothing methods escalates exponentially. Reducing the number of dimensions makes the operation computable. Dimension reduction aims to show only the most important directions of the data. SIR uses the inverse regression curve, ${\displaystyle E({\underline {x}}\,|\,y)}$ to perform a weighted principal component analysis, with which one identifies the effective dimension reducing directions.

This article first introduces the reader to the subject of dimension reduction and how it is performed using the model here. There is then a short review on inverse regression, which later brings these pieces together.

Model

Given a response variable ${\displaystyle \,Y}$ and a (random) vector ${\displaystyle X\in \mathbb {R} ^{p}}$ of explanatory variables, SIR is based on the model

${\displaystyle Y=f(\beta _{1}^{\top }X,\ldots ,\beta _{k}^{\top }X,\varepsilon )\quad \quad \quad \quad \quad (1)}$

where ${\displaystyle \beta _{1},\ldots ,\beta _{k}}$ are unknown projection vectors. ${\displaystyle \,k}$ is an unknown number (the dimensionality of the space we try to reduce our data to) and, of course, as we want to reduce dimension, smaller than ${\displaystyle \,p}$. ${\displaystyle \;f}$ is an unknown function on ${\displaystyle \mathbb {R} ^{k+1}}$, as it only depends on ${\displaystyle \,k}$ arguments, and ${\displaystyle \varepsilon }$ is the error with ${\displaystyle E[\varepsilon |X]=0}$ and finite variance ${\displaystyle \sigma ^{2}}$. The model describes an ideal solution, where ${\displaystyle \,Y}$ depends on ${\displaystyle X\in \mathbb {R} ^{p}}$ only through a ${\displaystyle \,k}$ dimensional subspace; i.e., one can reduce the dimension of the explanatory variables from ${\displaystyle \,p}$ to a smaller number ${\displaystyle \,k}$ without losing any information.

An equivalent version of ${\displaystyle \,(1)}$ is: the conditional distribution of ${\displaystyle \,Y}$ given ${\displaystyle \,X}$ depends on ${\displaystyle \,X}$ only through the ${\displaystyle \,k}$ dimensional random vector ${\displaystyle (\beta _{1}^{\top }X,\ldots ,\beta _{k}^{\top }X)}$. It is assumed that this reduced vector is as informative as the original ${\displaystyle \,X}$ in explaining ${\displaystyle \,Y}$.

The unknown ${\displaystyle \,\beta _{i}'s}$ are called the effective dimension reducing directions (EDR-directions). The space that is spanned by these vectors is denoted the effective dimension reducing space (EDR-space).

Relevant linear algebra background

To be able to visualize the model, note a short review on vector spaces:

For the definition of a vector space and some further properties I will refer to the article Linear Algebra and Gram-Schmidt Orthogonalization or any textbook in linear algebra and mention only the most important facts for understanding the model.

As the EDR-space is a ${\displaystyle \,k}$-dimensional subspace, we need to know what a subspace is. A subspace of ${\displaystyle \mathbb {R} ^{n}}$ is defined as a subset ${\displaystyle U\in \mathbb {R} ^{n}}$, if it holds that

${\displaystyle {\underline {a}},{\underline {b}}\in U\Rightarrow {\underline {a}}+{\underline {b}}\in U}$

${\displaystyle {\underline {a}}\in U,\lambda \in \mathbb {R} \Rightarrow \lambda {\underline {a}}\in U}$

Given ${\displaystyle {\underline {a}}_{1},\ldots ,{\underline {a}}_{r}\in \mathbb {R} ^{n}}$, then ${\displaystyle V:=L({\underline {a}}_{1},\ldots ,{\underline {a}}_{r})}$, the set of all linear combinations of these vectors, is called a linear subspace and is therefore a vector space. One says, the vectors ${\displaystyle {\underline {a}}_{1},\ldots ,{\underline {a}}_{r}}$ span ${\displaystyle \,V}$. But the vectors that span a space ${\displaystyle \,V}$ are not unique. This leads us to the concept of a basis and the dimension of a vector space:

A set ${\displaystyle B=\{{\underline {b}}_{1},\ldots ,{\underline {b}}_{r}\}}$ of linear independent vectors of a vector space ${\displaystyle \,V}$ is called basis of ${\displaystyle \,V}$, if it holds that

${\displaystyle V:=L({\underline {b}}_{1},\ldots ,{\underline {b}}_{r})}$

The dimension of ${\displaystyle \,V(\in \mathbb {R} ^{n})}$ is equal to the maximum number of linearly independent vectors in ${\displaystyle \,V}$. A set of ${\displaystyle \,n}$ linear independent vectors of ${\displaystyle \mathbb {R} ^{n}}$ set up a basis of ${\displaystyle \mathbb {R} ^{n}}$. The dimension of a vector space is unique, as the basis itself is not. Several bases can span the same space. Of course also dependent vectors span a space, but the linear combinations of the latter can give only rise to the set of vectors lying on a straight line. As we are searching for a ${\displaystyle \,k}$dimensional subspace, we are interested in finding ${\displaystyle \,k}$ linearly independent vectors that span the ${\displaystyle \,k}$dimensional subspace we want to project our data on.

Curse of dimensionality

The reason why we want to reduce the dimension of the data is due to the "curse of dimensionality" and of course, for graphical purposes. The curse of dimensionality is due to rapid increase in volume adding more dimensions to a (mathematical) space. For example, consider 100 observations from support ${\displaystyle [0,1]}$, which cover the interval quite well, and compare it to 100 observations from the corresponding ${\displaystyle 10}$ dimensional unit hypersquare, which are isolated points in a vast empty space. It is easy to draw inferences about the underlying properties of the data in the first case, whereas in the latter, it is not.

Inverse regression

Computing the inverse regression curve (IR) means instead of looking for

• ${\displaystyle \,E[Y|X=x]}$, which is a curve in ${\displaystyle \mathbb {R} ^{p}}$

we calculate

• ${\displaystyle \,E[X|Y=y]}$, which is also a curve in ${\displaystyle \mathbb {R} ^{p}}$, but consisting of ${\displaystyle \,p}$ one-dimensional regressions.

The center of the inverse regression curve is located at ${\displaystyle \,E[E[X|Y]]=E[X]}$. Therefore, the centered inverse regression curve is

• ${\displaystyle \,E[X|Y=y]-E[X]}$

which is a ${\displaystyle \,p}$ dimensional curve in ${\displaystyle \mathbb {R} ^{p}}$. In what follows we will consider this centered inverse regression curve and we will see that it lies on a ${\displaystyle \,k}$-dimensional subspace spanned by ${\displaystyle \,\Sigma _{xx}\beta _{i}\,'s}$.

But before seeing that this holds true, we will have a look at how the inverse regression curve is computed within the SIR-Algorithm, which will be introduced in detail later. What comes is the "sliced" part of SIR. We estimate the inverse regression curve by dividing the range of ${\displaystyle \,Y}$ into ${\displaystyle \,H}$ nonoverlapping intervals (slices), to afterwards compute the sample means ${\displaystyle \,{\hat {m}}_{h}}$ of each slice. These sample means are used as a crude estimate of the IR-curve, denoted as ${\displaystyle \,m(y)}$. There are several ways to define the slices, either in a way that in each slice are equally many observations, or we define a fixed range for each slice, so that we then get different proportions of the ${\displaystyle \,y_{i}\,'s}$ that fall into each slice.

Inverse regression versus dimension reduction

As just mentioned, the centered inverse regression curve lies on a ${\displaystyle \,k}$-dimensional subspace spanned by ${\displaystyle \,\Sigma _{xx}\beta _{i}\,'s}$ (and therefore also the crude estimate we compute). This is the connection between our Model and Inverse Regression. We shall see that this is true, with only one condition on the design distribution that must hold. This condition is, that:

${\displaystyle \forall \,{\underline {b}}\in \mathbb {R} ^{p}:\,E[b^{\top }X|\beta _{1}^{\top }X=\beta _{1}^{\top }x,\ldots ,\beta _{k}^{\top }X=\beta _{k}^{\top }x)=c_{0}+\sum _{i=1}^{k}c_{i}\beta _{i}^{\top }x}$

I.e. the conditional expectation is linear in ${\displaystyle \beta _{1}X,\ldots ,\beta _{k}X}$, that is, for some constants ${\displaystyle c_{0},\ldots ,c_{K}}$. This condition is satisfied when the distribution of ${\displaystyle \,X}$ is elliptically symmetric (e.g. the normal distribution). This seems to be a pretty strong requirement. It could help, for example, to closer examine the distribution of the data, so that outliers can be removed or clusters can be separated before analysis.

Given this condition and ${\displaystyle \,(1)}$, it is indeed true that the centered inverse regression curve ${\displaystyle \,E[X|Y=y]-E[X]}$ is contained in the linear subspace spanned by ${\displaystyle \,\Sigma _{xx}\beta _{k}(k=1,\ldots ,K)}$, where ${\displaystyle \,\Sigma _{xx}=Cov(X)}$. The proof is provided by Duan and Li in Journal of the American Statistical Association (1991).

Estimation of the EDR-directions

After having had a look at all the theoretical properties, our aim is now to estimate the EDR-directions. For that purpose, we conduct a (weighted) principal component analysis for the sample means ${\displaystyle \,{\hat {m}}_{h}\,'s}$, after having standardized ${\displaystyle \,X}$ to ${\displaystyle \,Z=\Sigma _{xx}^{-1/2}\{X-E(X)\}}$. Corresponding to the theorem above, the IR-curve ${\displaystyle \,m_{1}(y)=E[Z|Y=y]}$ lies in the space spanned by ${\displaystyle \,(\eta _{1},\ldots ,\eta _{k})}$, where ${\displaystyle \,\eta _{i}=\Sigma _{xx}^{1/2}\beta _{i}}$. (Due to the terminology introduced before, the ${\displaystyle \,\eta _{i}\,'s}$ are called the standardized effective dimension reducing directions.) As a consequence, the covariance matrix ${\displaystyle \,cov[E[Z|Y]]}$ is degenerate in any direction orthogonal to the ${\displaystyle \,\eta _{i}\,'s}$. Therefore, the eigenvectors ${\displaystyle \,\eta _{k}(k=1,\ldots ,K)}$ associated with the ${\displaystyle \,K}$ largest eigenvalues are the standardized EDR-directions.

Back to PCA. That is, we calculate the estimate for ${\displaystyle \,Cov\{m_{1}(y)\}}$:

${\displaystyle {\hat {V}}=n^{-1}\sum _{i=1}^{S}n_{s}{\bar {z}}_{s}{\bar {z}}_{s}^{\top }}$

and identify the eigenvalues ${\displaystyle {\hat {\lambda }}_{i}}$ and the eigenvectors ${\displaystyle {\hat {\eta }}_{i}}$ of ${\displaystyle {\hat {V}}}$, which are the standardized EDR-directions. (For more details about that see next section: Algorithm.) Remember that the main idea of PC transformation is to find the most informative projections that maximize variance!

Note that in some situations SIR does not find the EDR-directions. One can overcome this difficulty by considering the conditional covariance ${\displaystyle \,Cov(X|Y)}$. The principle remains the same as before, but one investigates the IR-curve with the conditional covariance instead of the conditional expectation. For further details and an example where SIR fails, see Härdle and Simar (2003).

Algorithm

The algorithm to estimate the EDR-directions via SIR is as follows. It is taken from the textbook Applied Multivariate Statistical Analysis (Härdle and Simar 2003)

1. Let ${\displaystyle \,\Sigma _{xx}}$ be the covariance matrix of ${\displaystyle \,X}$. Standardize ${\displaystyle \,X}$ to

${\displaystyle \,Z=\Sigma _{xx}^{-1/2}\{X-E(X)\}}$

(We can therefore rewrite ${\displaystyle \,(1)}$ as

${\displaystyle Y=f(\eta _{1}^{\top }Z,\ldots ,\eta _{k}^{\top }Z,\varepsilon )}$

where ${\displaystyle \,\eta _{k}=\beta _{k}\Sigma _{xx}^{1/2}\quad \forall \;k}$ For the standardized variable Z it holds that ${\displaystyle \,E[Z]=0}$ and ${\displaystyle \,Cov(Z)=I}$.)

2. Divide the range of ${\displaystyle \,y_{i}}$ into ${\displaystyle \,S}$ nonoverlapping slices ${\displaystyle \,H_{s}(s=1,\ldots ,S).\;n_{s}}$ is the number of observations within each slice and ${\displaystyle \,I_{H_{s}}}$ the indicator function for this slice:

${\displaystyle n_{s}=\sum _{i=1}^{n}I_{H_{s}}(y_{i})}$

3. Compute the mean of ${\displaystyle \,z_{i}}$ over all slices, which is a crude estimate ${\displaystyle \,{\hat {m}}_{1}}$ of the inverse regression curve ${\displaystyle \,m_{1}}$:

${\displaystyle \,{\bar {z}}_{s}=n_{s}^{-1}\sum _{i=1}^{n}z_{i}I_{H_{s}}(y_{i})}$

4. Calculate the estimate for ${\displaystyle \,Cov\{m_{1}(y)\}}$:

${\displaystyle \,{\hat {V}}=n^{-1}\sum _{i=1}^{S}n_{s}{\bar {z}}_{s}{\bar {z}}_{s}^{\top }}$

5. Identify the eigenvalues ${\displaystyle \,{\hat {\lambda }}_{i}}$ and the eigenvectors ${\displaystyle \,{\hat {\eta }}_{i}}$ of ${\displaystyle \,{\hat {V}}}$, which are the standardized EDR-directions.

6. Transform the standardized EDR-directions back to the original scale. The estimates for the EDR-directions are given by:

${\displaystyle \,{\hat {\beta }}_{i}={\hat {\Sigma }}_{xx}^{-1/2}{\hat {\eta }}_{i}}$

(which are not necessarily orthogonal)

For examples, see the book by Härdle and Simar (2003).

References

• Li, K-C. (1991) "Sliced Inverse Regression for Dimension Reduction", Journal of the American Statistical Association, 86, 316–327 Jstor
• Cook, R.D. and Sanford Weisberg, S. (1991) "Sliced Inverse Regression for Dimension Reduction: Comment", Journal of the American Statistical Association, 86, 328–332 Jstor
• Härdle, W. and Simar, L. (2003) Applied Multivariate Statistical Analysis, Springer Verlag. ISBN 3-540-03079-4
• Kurzfassung zur Vorlesung Mathematik II im Sommersemester 2005, A. Brandt