Cross-validation on AutoML models across multiple datasets not working properly

[egolub22]

I am trying to create a program that will run hyper-parameter tuning of the AutoML models on a dataframe that contains multiple datasets (unique_ids). However, since it appears that the AutoMLForecast class has no cross_validation function, I am creating an MLForecast object for each trained model and crossvalidating on the original dataframe. The issue is that the accuracy metrics suggests that the forecasts for all but one of the datasets are incredibly poor. This leads me to believe that the correct models are not being applied to the correct datasets.

I'm looking for the best way to cross-validate AutoML models across multiple datasets

`
Cell 1:

transformed_df, future_df = fourier(df, freq='D', season_length=365, k=12, h=31)
transformed_df

optuna.logging.set_verbosity(optuna.logging.ERROR)


def my_init_config(trial: optuna.Trial):

    lag_trial_1 = trial.suggest_categorical('lag_trial_1', [1, 7, 365])
    lag_trial_2 = trial.suggest_categorical('lag_trial_2', [1, 7, 365])
    rollingstd_window_trial = trial.suggest_categorical('rollingstd_window_trial', [2, 7, 31])
    rollingmean_window_trial = trial.suggest_categorical('rollingmean_window_trial', [2, 7, 31])

    return {
        'date_features': ['dayofweek', 'month', 'dayofyear'],
        'lags': [1, 7, 365], 
            'lag_transforms': {lag_trial_1: [RollingStd(window_size=rollingstd_window_trial)], lag_trial_2: [RollingMean(window_size=rollingmean_window_trial)]},
        }
    models_full = {'RandomForest': AutoRandomForest(), 'ElasticNet': AutoElasticNet(), 'Lasso': AutoLasso(), 'LinearRegression': AutoLinearRegression(), 'Catboost': AutoCatboost(), 'XGBoost': AutoXGBoost(), 'LightGBM': AutoLightGBM(), 'Ridge': AutoRidge()}
    models_single = {'RandomForest': AutoRandomForest()}
    
    auto_mlf = AutoMLForecast( 
        models=models_full,
        freq="D",
        season_length=7,
        init_config=my_init_config,
        num_threads=72
    )

horizon = 31
start_time = time.perf_counter()
auto_mlf.fit(
    transformed_df,
    n_windows=12,
    h=horizon,
    num_samples=10,  # number of trials to run
)
end_time = time.perf_counter()
print(f"finished, Time Elapsed:{end_time - start_time:0.4f}")

`

`
Cell 2:

start_time = time.perf_counter()
for i, model in enumerate(auto_mlf.results_.keys()):
    print(i)
    print()
    model_mlf = auto_mlf.models_[model]
    model_cv = model_mlf.cross_validation(transformed_df, n_windows=12, h=31)

    if(i == 0):
        ml_cv = model_cv
    else:
        ml_cv[model] = model_cv[model]

end_time = time.perf_counter()
print(f"finished, Time Elapsed:{end_time - start_time:0.4f}")

ml_cv

`

[jmoralez]

How hard is it to see which library you're using and search for the corresponding repo?

[egolub22]

My fault

[egolub22]

Any suggestions though about this issue

[jmoralez]

This leads me to believe that the correct models are not being applied to the correct datasets.

What do you mean by this? A single model is trained on all series.

The auto models run cross validation internally and return the average metric across folds to the optimization process, so the best hyperparameters will be chosen based on that, if the default metric (SMAPE) isn't suited for your use case you should use a different one.