Utilforecast Evaluate method generating error 'models'

[egolub22]

I am trying to use the evaluate method to get accuracy numbers for several models using cross validation, however, I cannot get the method to run without giving the error shown below. I have confirmed that the 'models' variable given as an argument is in fact a list of model names, all of which are available as columns in the 'crossvalidation_df' dataframe.

`
Cell 1:

sf = StatsForecast(
    models=[AutoARIMA(season_length=7),
           AutoETS(season_length=7),
           AutoCES(season_length=7),
           AutoTheta(season_length=7),
           Theta(season_length=7),
           OptimizedTheta(season_length=7),
           DynamicTheta(season_length=7),
           DynamicOptimizedTheta(season_length=7),
           MSTL(season_length = [7, 365]),
           SeasonalExponentialSmoothingOptimized(season_length  = 7),
           ],
    freq='D',
    n_jobs=-1
)

`

`
Cell 2:

crossvalidation_df = sf.cross_validation(
    df=df,
    h=30,
    step_size=30,
    n_windows=2
)

`

`
Cell 3:

warnings.simplefilter(action='ignore', category=FutureWarning)

model_num = 10
window_num = 2
step_size = 30
graph_range = 365
models = crossvalidation_df.drop(columns=['ds', 'cutoff', 'y']).columns.tolist()

fig = make_subplots(rows=model_num*window_num, cols=1, subplot_titles=[" " for x in range(model_num*window_num)])

evals = []
for j, cutoff in enumerate(crossvalidation_df['cutoff'].unique()):

    eval_ = evaluate(crossvalidation_df[crossvalidation_df['cutoff'] == cutoff], metrics=[rmse], models=models)
    evals.append(eval_)
    
    for i in range(3,model_num+3):

        cv_ind = i-2+(j*model_num)

        y_val = crossvalidation_df[crossvalidation_df['cutoff'] == cutoff]["y"]
        y_pred = crossvalidation_df[crossvalidation_df['cutoff'] == cutoff].iloc[:, i]
        ds_pred = crossvalidation_df[crossvalidation_df['cutoff'] == cutoff]["ds"]
        
        mape_var = mape(y_val, y_pred) * 100
        mae_var = mae(y_val, y_pred)
        rmse_var = rmse(y_val, y_pred)
        relative_mae_var = mae(y_val, y_pred)/np.mean(y_val)
        monthly_ape_var = (np.abs(y_pred.sum(axis=0) - y_val.sum(axis=0))/y_val.sum(axis=0)) * 100
        
        max_val = np.max(df["y"][-graph_range:])
        if(np.max(y_pred) > max_val):
            max_val = np.max(y_pred)
    
        fig.append_trace(go.Scatter(x=df["ds"][-graph_range:], y=df["y"][-graph_range:], mode='lines', name='Sales (Base)', line={"width":1.3, "color":"black"}, legendgroup = cv_ind), row=cv_ind, col=1)
        fig.add_trace(go.Scatter(x=ds_pred, y=y_pred, mode='lines', name=crossvalidation_df.columns[i], line={"width":1.3, "color":"blue"}, legendgroup = cv_ind), row=cv_ind, col=1)
        fig.add_annotation(text=f'MAPE: {mape_var:.2f}%<br>MAE: {mae_var:.2f}<br>RMSE: {rmse_var:.2f}<br>Relative MAE: {relative_mae_var:.2f}<br>Monthly APE: {monthly_ape_var:.2f}%', 
                        align='left',
                        showarrow=False,
                        xref=f'x{cv_ind}',
                        yref=f'y{cv_ind}',
                        x="2023-07-01",
                        y=max_val,
                        bordercolor='red',
                        borderwidth=1)
    
        fig.layout.annotations[cv_ind-1]['text'] = (f"{crossvalidation_df.columns[i]} Sales Forecast between {ds_pred[0]} and {ds_pred[-1]}")

fig.update_layout(height=model_num*window_num*500, width=1300, title_text="Forecasts of Sales Data", legend_tracegroupgap=465.35)
fig.show()

`

The error:

[jmoralez]

What's the error that you get?

[egolub22]

I have edited to add the error, thanks for reminding me

[jmoralez]

Which rmse are you using?

[egolub22]

I have solved this issue, you were on the right track. I was not using a version of rmse from utilsforcast but instead from another package. Thanks