PB - Percentage Better

Calculates the percentage of cases where forecast error is greater than the benchmark model.

 

Syntax

PB(X, F, M, Mode)

X is the eventual outcome time series sample data (a one-dimensional array of cells e.g. row or column).

F is the forecast time series data (a one-dimensional array of cells e.g. row or column).

M is the seasonal period (for non-seasonal time series, set M=1 (default) or leave it blank).

Mode is a switch to select the basis of the calculated PB (0=regular error (default), 1=MAE, 2=MSE)

Mode Description
0 Relative Absolute Error (default)
1 Mean Absolute Error (MAE)
2 Mean Squared Error (MSE)
 

Remarks

  1. The time series is homogeneous or equally spaced.
  2. The time series X and F must be of identical size
  3. The time series X or F may include observations with missing values (e.g. #N/A or blank).
  4. Observations with missing values in Y or F are excluded from the MASE calculation.
  5. The regular Percentage Better (PB) is calculated by following formula:

    $$ \mathrm{PB}=\left\{\begin{array}{l} {\frac{\sum_{t=2}^N I\{\left | e_t \right | \lt \left | e_t^* \right | \}}{N-1}} \\ {\frac{\sum_{t=M+1}^N I\{\left | e_t \right | \lt \left | e_t^* \right | \}}{N-M}} \end{array}\right. \begin{array}{l} \{y_i\}\,\mathrm{is\,non-seasonal} \\ \{y_i\}\,\mathrm{is\,seasonal} \end{array} $$
    Where:
    • $\{e_t^*\}$ is the forecast error of the naïve benchmark model calculated in-sample.
    • $\{e_i\}$ is the forecast error of the actual forecast.
    • $I\{.\}$ is an operator that yields the values of zero or one, in accordance with the expression:
      $$ I(e_t)=\left\{\begin{array}{l} 1\\ 0 \end{array}\right. \begin{array}{l} \left | e_t \right | \gt \left | e_t^* \right | \\ \left | e_t \right | \leq \left | e_t^* \right | \end{array} $$
  6. In short, PB demonstrates the average number of times that forecasting method overcomes the naïve forecasting method
  7. The Percentage Better measure counts the number of cases when the forecasting model is superior to the naïve benchmark, but it does not evaluate the degree in which they are different
  8. PB does not indicate the amount of possible improvement. Thus, it is possible to have one method that performs slightly better than the benchmark method for 99 series but much worse on 1 series, giving that method a PB score of 99 even though the benchmark method is preferable.
  9. The close variant of PB is based on measures other than absolute forecast error. NumXL PB function supports two measures: MAE and MSE. The Percentage Better of MAE is calculated by the following formula:

    $$ \mathrm{PB(MAE)}=100\%\, \times\,\mathrm{mean(}I\{\mathrm{MAE}\lt \mathrm{MAE^*}\}\mathrm{)} $$
    $$ \mathrm{PB(MAE)}=100\%\, \times\,\mathrm{mean(}I\{\mathrm{MSE}\lt \mathrm{MSE^*}\}\mathrm{)} $$
  10. The PB function is available starting with version 1.65 HAMMOCK.

Examples

Example 1:

 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
A B C
Date Data Forecast
2008-01-01 -2.9 -2.95
2008-02-01 -2.83 -2.7
2008-03-01 -0.95 -1.00
2008-04-01 -0.88 -0.68
2008-05-01 1.21 1.50
2008-06-01 -1.67 -1.00
2008-07-01 0.83 0.90
2008-08-01 -0.27 -0.37
2008-09-01 1.36 1.26
2008-10-01 -0.34 -0.54
2008-11-01 0.48 0.58
2008-12-01 -2.83 -2.13
2009-01-01 -0.95 -0.75
2009-02-01 -0.88 -0.89
2009-03-01 1.21 1.25
2009-04-01 -1.67 -1.65
2009-05-01 -2.99 -3.20
2009-06-01 1.24 1.29
2009-07-01 0.64 0.60


  Formula Description (Result)
  =PB($B$3:$B$21,$C$3:$C$21,1,0) MASE (88.89%)
  =PB($B$3:$B$21,$C$3:$C$21,1,1) MASE (94.44%)
  =PB($B$3:$B$21,$C$3:$C$21,1,2) MASE (94.44%)

Files Examples

References

Have more questions? Submit a request

0 Comments