Calculates the percentage of cases where forecast error is greater than the benchmark model.
Syntax
PB(X, F, M, Mode)
- X
- is the eventual outcome time series sample data (a one-dimensional array of cells e.g. row or column).
- F
- is the forecast time series data (a one-dimensional array of cells e.g. row or column).
- M
- is the seasonal period (for non-seasonal time series, set M=1 (default) or leave it blank).
- Mode
- is a switch to select the basis of the calculated PB (0=regular error (default), 1=MAE, 2=MSE)
Mode Description 0 Relative Absolute Error (default) 1 Mean Absolute Error (MAE) 2 Mean Squared Error (MSE)
Remarks
- The time series is homogeneous or equally spaced.
- The time series X and F must be of identical size
- The time series X or F may include observations with missing values (e.g. #N/A or blank).
- Observations with missing values in Y or F are excluded from the MASE calculation.
- The regular Percentage Better (PB) is calculated by following formula:
$$ \mathrm{PB}=\left\{\begin{array}{l} {\frac{\sum_{t=2}^N I\{\left | e_t \right | \lt \left | e_t^* \right | \}}{N-1}} \\ {\frac{\sum_{t=M+1}^N I\{\left | e_t \right | \lt \left | e_t^* \right | \}}{N-M}} \end{array}\right. \begin{array}{l} \{y_i\}\,\mathrm{is\,non-seasonal} \\ \{y_i\}\,\mathrm{is\,seasonal} \end{array} $$
Where:
- $\{e_t^*\}$ is the forecast error of the naïve benchmark model calculated in-sample.
- $\{e_i\}$ is the forecast error of the actual forecast.
- $I\{.\}$ is an operator that yields the values of zero or one, in accordance with the expression:
$$ I(e_t)=\left\{\begin{array}{l} 1\\ 0 \end{array}\right. \begin{array}{l} \left | e_t \right | \gt \left | e_t^* \right | \\ \left | e_t \right | \leq \left | e_t^* \right | \end{array} $$
- In short, PB demonstrates the average number of times that forecasting method overcomes the naïve forecasting method
- The Percentage Better measure counts the number of cases when the forecasting model is superior to the naïve benchmark, but it does not evaluate the degree in which they are different
- PB does not indicate the amount of possible improvement. Thus, it is possible to have one method that performs slightly better than the benchmark method for 99 series but much worse on 1 series, giving that method a PB score of 99 even though the benchmark method is preferable.
- The close variant of PB is based on measures other than absolute forecast error. NumXL PB function supports two measures: MAE and MSE. The Percentage Better of MAE is calculated by the following formula:
$$ \mathrm{PB(MAE)}=100\%\, \times\,\mathrm{mean(}I\{\mathrm{MAE}\lt \mathrm{MAE^*}\}\mathrm{)} $$
$$ \mathrm{PB(MAE)}=100\%\, \times\,\mathrm{mean(}I\{\mathrm{MSE}\lt \mathrm{MSE^*}\}\mathrm{)} $$ - The PB function is available starting with version 1.65 HAMMOCK.
Examples
Example 1:
|
|
| Formula | Description (Result) |
|---|---|
| =PB($B$3:$B$21,$C$3:$C$21,1,0) | MASE (88.89%) |
| =PB($B$3:$B$21,$C$3:$C$21,1,1) | MASE (94.44%) |
| =PB($B$3:$B$21,$C$3:$C$21,1,2) | MASE (94.44%) |
Files Examples
Related Links
References
- R.J. Hyndman, A.B. Koehler, "Another look at measures of forecast accuracy", International Journal of Forecasting, 22 (2006), pp. 679-688
- James Douglas Hamilton; Time Series Analysis; Princeton University Press; 1st edition(Jan 11, 1994), ISBN: 691042896
- Tsay, Ruey S.; Analysis of Financial Time Series; John Wiley & SONS; 2nd edition(Aug 30, 2005), ISBN: 0-471-690740
- D. S.G. Pollock; Handbook of Time Series Analysis, Signal Processing, and Dynamics; Academic Press; Har/Cdr edition(Nov 17, 1999), ISBN: 125609906
Comments
Article is closed for comments.