Feature Request: Time Series Forecasting Datasets & Task Definitions

[shchur]

Motivation

Time series forecasting is an extremely popular machine learning task. However, there doesn't exist a unified repository of forecasting datasets & task definitions. OpenML has an opportunity to fill this gap, which would be very extremely helpful for researchers working in this area.

Prior work

As far as I know, the only successful existing attempt at constructing such a repository of forecasting datasets has been made by the Monash Forecasting Repository. However it has several limitations:

• it is not actively maintained (last activity >1 year ago)
• it cannot be extended by external contributors
• it only provides datasets and not the respective task definitions, which makes it difficult to use it as a unified benchmark
• it lacks basic infrastructure like efficient dataset loaders and evaluation scripts

Time series forecasting

We considered the following setting:

1. point forecast - we predict a single value for each of the future time step (same as in tabular regression).
   
2. the dataset contains multiple univariate time series - the time series are not assumed to be aligned in time, each time series can have different length.

Dataset schema

A time series dataset can be represented as a tabular dataset with some additional constraints on the schema.

At the very minimum, the dataset should contain a single table with 3 columns:

• item_id: Unique identifier of each time series
• timestamp: Date & time when the measurement was recorded
• target: Time series value that needs to be predicted

For example, a dataset with 2 time series, each containing 3 observations made at daily frequency can be represented as

item_id	timestamp	target
A	2023-01-01	0.5
A	2023-01-02	2.5
A	2023-01-03	1.2
B	2020-05-10	3.0
B	2020-05-11	1.4
B	2020-05-12	0.0

Such tabular representation for time series is rather standard and is used by many open-source packages (e.g., Prophet, StatsForecast) and Kaggle competitions (e.g., M5, Corporación Favorita Grocery Sales Forecasting ).

The dataset may contain additional columns corresponding to so-called covariates. For example, when predicting sales for different products, a related covariate may represent the price of each product at a given day.

Task definition

At the very minimum, the task definition should contain

• prediction_length (a.k.a. forecast_horizon): an integer that denotes how many future values need to be predicted for each individual time series in the dataset.

Train/test split

In time series forecasting, the train/test split cannot be decoupled from the task definition (as in tabular tasks). For example, given the above data and prediction_length=1, the train set data_train would correspond to

item_id	timestamp	target
A	2023-01-01	0.5
A	2023-01-02	2.5
B	2020-05-10	3.0
B	2020-05-11	1.4

and the test set data_test would contain

item_id	timestamp	target
A	2023-01-03	1.2
B	2020-05-12	0.0

Each forecasting algorithm needs to produce predictions predictions in the following format

item_id	timestamp	forecast
A	2023-01-03	1.0
B	2020-05-12	2.0

Note that some time series metrics depend on the historic data. That is, they are defined not as metric(data_test, predictions), but rather as metric(data_test, predictions, data_train) (e.g., MASE).

Potential extensions

• Probabilistic forecasting (a.k.a. quantile forecasting) - instead of predicting a single value for each future time step (as in point forecast), we predict multiple quantiles. This is similar to the difference between tabular regression and tabular quantile regression.
  
• Dataset schema - we may include additional information such as static covariates or past-only covariates (see here for an example)
• Task description - we may include additional information, such as what metric should be optimized.

Instances of forecasting problems

Kaggle competitions

• M5
• Corporación Favorita Grocery Sales Forecasting
• Restaurant Visitor Forecasting
• GoDaddy - Microbusiness Density Forecasting

Scientific publications

• https://arxiv.org/abs/2012.07436
• https://arxiv.org/abs/1704.04110

[PGijsbers]

Thanks for the proposal and taking time to illustrate the task and provide references. We are currently rewriting the server back-end so we have a hold on adding new task types, though we hope to be able to start adding task types again sometime early next year. Nevertheless, that shouldn't stop us from further discussing the task type and seeing whether we would like to adopt it when we are ready. I'll have a closer look at this later 👍