In order to be able to assess the quality of the constructed models, as well as to make a decision on the choice of a particular behavior strategy, efficiency criteria are needed. A number of criteria have been selected for this system. All criteria are implemented in the test block and are calculated on test samples so that the assessment is the most relevant. The first criterion that directly shows the deviation itself is the mean absolute deviation. This criterion shows the average error for the modulo model. With an increase in the value of this indicator, the relevance of the forecasts of a particular model is lost. The calculation formula looks like: where is the real value of the indicator, and is predicted by the system.
The rationale for choosing this criterion is that, with its simplicity, this criterion provides sufficiently relevant data on the behavior of the system. The next criterion that was chosen to check the quality of the constructed models is the average error percentage. This indicator, like the previous criterion, reflects the deviation from real data, except that this indicator is expressed as a percentage. This approach will allow better evaluation and comparison of models. Moreover, this indicator allows you to evaluate the same model, built and tested on different data. Also, this criterion allows you to compare models with each other. The calculation formula looks like: where is the real value of the indicator, and is predicted by the system.
The average absolute percentage of errors is another universal criterion for comparing models with each other. Due to the fact that the error is expressed as a percentage, we can compare the quality of work on different time series for one stock and on the same time series for different stocks. This approach allows you to identify the strengths and weaknesses of the models and choose the most suitable for a particular case. The formula looks like: where is the real value of the indicator, and is predicted by the system. The standard deviation from the mean is a fairly common criterion in statistics. This value shows how much the values in the set deviate from the average for the given set. This indicator can be called a measure of uncertainty in the sense that it displays the spread of values.
The higher this indicator, the correspondingly greater the spread. The formula for this indicator is: where are the values of the set, and is the average value over the entire set. Another criterion that is used to assess the quality of models is the coefficient of determination. R^2 is the proportion of the variance of the variable being explained that is modeled using this set of variables. In other words, the main meaning of this criterion lies in the fact that the closer the value is to 1, the better the constructed model fits the data set. For the adequacy of the model, the value of the indicator should be greater than 0.5.
The formula for calculating this coefficient is: where is the variance of the random error of the model, is the variance of the dependent variable. The following three coefficients apply only to autoregressive models. The first such criterion in the system is the Akaike information criterion (AIC). This criterion applies only to the evaluation of statistical models. Moreover, the statistical model should be built using the maximum likelihood method. The formula for calculating this criterion in the general case is: where k is the number of model parameters, and L is the maximum likelihood function. AIC can also be expressed in terms of the number of observations and the residual sum of squares, in which case the formula becomes: where n is the number of observations and RSS is the residual sum of squares.
Another information criterion used to evaluate and compare autoregressive models is the Bayesian Information Criterion (BIC). Unlike the AIC criterion, the BIC imposes a penalty for the complexity of the model, in other words, for increasing the number of explanatory variables. The BIC formula looks like: Another criterion, which is analogous to AIC and BIC, is called the Hanna-Queen criterion (HQC). This criterion is also used to select the best model. 3.2 Methodology for the study of efficiency In this work, the system will be tested on the shares of two companies belonging to the information technology industry, namely Microsoft. The analysis will be carried out at two time intervals (Table 3), taking into account the limitation imposed by the test block.
The time intervals for analysis are presented in the table. The analysis will be carried out at the closing price. Table 3. Time frames for analysis. Time period 1 Time period 22013/1/1 - 2016/12/312015/1/1 - 2016/12/31 Time periods differ in their duration: four years and two years. From the test sets presented in the system for making forecasts, two sets were selected - two and four months long. For each company for each time period, a full-fledged analysis was carried out with the calculation of the above efficiency criteria.
sleep Chala is the construction of all moving averages. Since these are just smoothing lines that do not depend on the size of the training sample, the moving data is built for the second time period. Simple, triangular and exponential moving averages are built for three different periods: fast (10), medium (50) and slow (100). Such a choice of periods is due to the fact that the definition of a trend depends on the intersection or relative position of moving averages of different periods. Our system checks the correctness of this statement. For each moving average of all the above, three charts are plotted - a time series chart, a difference chart, and a trend chart. For moving averages, the MAE, MPE, MAPE and MSE criteria are still being calculated.
These criteria will allow you to evaluate the quality of a particular moving average and compare their types. As for the double and triple exponential moving averages, for them, in addition to the periods of 10, 50, 100, a number of parameters beta (for beta [0.25;0.5]) and gamma (for gamma [0.05;0.1]) were chosen, and with the help of graphs and performance criteria, we will have the opportunity to determine the best option with indicators. Next, modeling takes place using discriminant analysis methods: logistic regression, linear and quadratic discriminant analyses.
First, the system compares two types of input data: binary and quantitative. There is also a search for the optimal number of variables for analysis, taking into account the minimization of the number of parameters and the maximization of indicators. The system considers several options: 2, 5, 10, 25. Then, to check the values, predictions are made on the test data. To compare and evaluate the results, the MPE indicators are calculated for the training and test samples. .