Mean absolute percentage error

The mean absolute percentage error (MAPE), also known as mean absolute percentage deviation (MAPD), is a measure of prediction accuracy of a forecasting method in statistics, for example in trend estimation, also used as a loss function for regression problems in machine learning. It usually expresses the accuracy as a ratio defined by the formula:

{\mbox{M}}={\frac {1}{n}}\sum _{t=1}^{n}\left|{\frac {A_{t}-F_{t}}{A_{t}}}\right|,

where $A t$ is the actual value and $F t$ is the forecast value. The MAPE is also sometimes reported as a percentage, which is the above equation multiplied by 100. The difference between $A t$ and $F t$ is divided by the actual value $A t$ again. The absolute value in this calculation is summed for every forecasted point in time and divided by the number of fitted points $n$ . Multiplying by 100% makes it a percentage error.

MAPE in regression problems

Mean absolute percentage error is commonly used as a loss function for regression problems and in model evaluation, because of its very intuitive interpretation in terms of relative error.

Definition

Consider a standard regression setting in which the data are fully described by a random pair $Z=(X,Y)$ with values in $\mathbb {R} ^{d}\times \mathbb {R}$ , and $n$ i.i.d. copies $(X_{1},Y_{1}),...,(X_{n},Y_{n})$ of $(X,Y)$ . Regression models aims at finding a good model for the pair, that is a measurable function $g$ from $\mathbb {R} ^{d}$ to $\mathbb {R}$ such that $g(X)$ is close to $Y$ .

In the classical regression setting, the closeness of $g(X)$ to $Y$ is measured via the $L 2$ risk, also called the mean squared error (MSE). In the MAPE regression context,[1] the closeness of $g(X)$ to $Y$ is measured via the MAPE, and the aim of MAPE regressions is to find a model $g_{\text{MAPE}}$ such that:

g_{\text{MAPE}}(x)=\arg \min _{g\in {\mathcal {G}}}\mathbb {E} \left[\left|{\frac {g(X)-Y}{Y}}\right||X=x\right]

where ${\mathcal {G}}$ is the class of models considered (e.g. linear models).

In practice

In practice $g_{\text{MAPE}}(x)$ can be estimated by the empirical risk minimization strategy, leading to

{\widehat {g}}_{\text{MAPE}}(x)=\arg \min _{g\in {\mathcal {G}}}\sum _{i=1}^{n}\left|{\frac {g(X_{i})-Y_{i}}{Y_{i}}}\right|

From a practical point of view, the use of the MAPE as a quality function for regression model is equivalent to doing weighted mean absolute error (MAE) regression, also known as quantile regression. This property is trivial since

{\widehat {g}}_{\text{MAPE}}(x)=\arg \min _{g\in {\mathcal {G}}}\sum _{i=1}^{n}\omega (Y_{i})\left|g(X_{i})-Y_{i}\right|{\mbox{ with }}\omega (Y_{i})=\left|{\frac {1}{Y_{i}}}\right|

As a consequence, the use of the MAPE is very easy in practice, for example using existing libraries for quantile regression allowing weights.

Consistency

The use of the MAPE as a loss function for regression analysis is feasible both on a practical point of view and on a theoretical one, since the existence of an optimal model and the consistency of the empirical risk minimization can be proved.[1]

Alternative MAPE definitions

Problems can occur when calculating the MAPE value with a series of small denominators. A singularity problem of the form 'one divided by zero' and/or the creation of very large changes in the Absolute Percentage Error, caused by a small deviation in error, can occur.

As an alternative, each actual value ( $A t$ ) of the series in the original formula can be replaced by the average of all actual values ( $Ā t$ ) of that series. This alternative is still being used for measuring the performance of models that forecast spot electricity prices.[2]

Note that this is equivalent to dividing the sum of absolute differences by the sum of actual values, and is sometimes referred to as WAPE (weighted absolute percentage error).

Issues

Although the concept of MAPE sounds very simple and convincing, it has major drawbacks in practical application,[3] and there are many studies on shortcomings and misleading results from MAPE.[4][5]

It cannot be used if there are zero values (which sometimes happens for example in demand data) because there would be a division by zero.
For forecasts which are too low the percentage error cannot exceed 100%, but for forecasts which are too high there is no upper limit to the percentage error.
MAPE puts a heavier penalty on negative errors, $A_{t}<F_{t}$ than on positive errors.[6] As a consequence, when MAPE is used to compare the accuracy of prediction methods it is biased in that it will systematically select a method whose forecasts are too low. This little-known but serious issue can be overcome by using an accuracy measure based on the logarithm of the accuracy ratio (the ratio of the predicted to actual value), given by $\log \left({\frac {\text{predicted}}{\text{actual}}}\right)$ . This approach leads to superior statistical properties and leads to predictions which can be interpreted in terms of the geometric mean.[3]

To overcome these issues with MAPE, there are some other measures proposed in literature:

Mean Absolute Scaled Error (MASE)
Symmetric Mean Absolute Percentage Error (sMAPE)
Mean Directional Accuracy (MDA)
Mean Arctangent Absolute Percentage Error (MAAPE): MAAPE is a new metric of absolute percentage error, and has been developed through looking at MAPE from a different angle. In essence, MAAPE is a slope as an angle, while MAPE is a slope as a ratio.[5]

External links

References

de Myttenaere, B Golden, B Le Grand, F Rossi (2015). "Mean absolute percentage error for regression models", Neurocomputing 2016 arXiv:1605.02541
Jorrit Vander Mynsbrugge (2010). "Bidding Strategies Using Price Based Unit Commitment in a Deregulated Power Market", K.U.Leuven
Tofallis (2015). "A Better Measure of Relative Prediction Accuracy for Model Selection and Model Estimation", Journal of the Operational Research Society, 66(8):1352-1362. archived preprint
Hyndman, Rob J., and Anne B. Koehler (2006). "Another look at measures of forecast accuracy." International Journal of Forecasting, 22(4):679-688 doi:10.1016/j.ijforecast.2006.03.001.
Kim, Sungil and Heeyoung Kim (2016). "A new metric of absolute percentage error for intermittent demand forecasts." International Journal of Forecasting, 32(3):669-679 doi:10.1016/j.ijforecast.2015.12.003.
Makridakis, Spyros (1993) "Accuracy measures: theoretical and practical concerns." International Journal of Forecasting, 9(4):527-529 doi:10.1016/0169-2070(93)90079-3

This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.

[demyttenaere2016-1] Myttenaere, B Golden, B Le Grand, F Rossi (2015). "Mean absolute percentage error for regression models", Neurocomputing 2016 arXiv:1605.02541

[2] Jorrit Vander Mynsbrugge (2010). "Bidding Strategies Using Price Based Unit Commitment in a Deregulated Power Market", K.U.Leuven

[tofallis2015-3] Tofallis (2015). "A Better Measure of Relative Prediction Accuracy for Model Selection and Model Estimation", Journal of the Operational Research Society, 66(8):1352-1362. archived preprint

[4] Hyndman, Rob J., and Anne B. Koehler (2006). "Another look at measures of forecast accuracy." International Journal of Forecasting, 22(4):679-688 doi:10.1016/j.ijforecast.2006.03.001.

[Kim2016-5] Kim, Sungil and Heeyoung Kim (2016). "A new metric of absolute percentage error for intermittent demand forecasts." International Journal of Forecasting, 32(3):669-679 doi:10.1016/j.ijforecast.2015.12.003.

[6] Makridakis, Spyros (1993) "Accuracy measures: theoretical and practical concerns." International Journal of Forecasting, 9(4):527-529 doi:10.1016/0169-2070(93)90079-3