Missing multiclass-multioutput support

[mitar]

I see that multiclass-multioutput support is missing? Couldn't in that case code just split learning into one model per output?

[mfeurer]

Yes, that is correct. The feature is not implemented since there is no metric for that in scikit-learn, no one of the contributors so far needed that feature and we do not have any data to test that. Do you have any reference for multiclass-multioutput?

[mitar]

I think my issue was in fact #293. A continuous-multioutput was misclassified as multiclass-multioutput so this is why I thought I need it.

I see that there is an active pull requests to add a metric though: scikit-learn/scikit-learn#3681 (addressing scikit-learn/scikit-learn#3453) But it looks stale.

[mfeurer]

Okay, I see. However, continuous-multioutput is not supported by auto-sklearn. This type of class is only supported by the tree-based models in scikit-learn, right? What would be an application of this kind of data?

[mitar]

No, it seems it is supported by any regressor:

Multioutput regression support can be added to any regressor with MultiOutputRegressor. This strategy consists of fitting one regressor per target. Since each target is represented by exactly one regressor it is possible to gain knowledge about the target by inspecting its corresponding regressor. As MultiOutputRegressor fits one regressor per target it can not take advantage of correlations between targets.

BTW, similar can be done for classifiers as well, using MultiOutputClassifier.

I think it would be a good first start, just to have something.

What would be an application of this kind of data?

I mean, application is that I can run auto-sklearn on any dataset I get, no matter the task it has. ;-)

[mfeurer]

The reason why we didn't implement anything together with the MultiOutputClassifier and MultiOutputRegressor is that they didn't nicely support partial_fit and models with warmstarts. As partial_fit seems to be added in sklearn==0.19 we can hopefully add this feature then.

[BrechtBa]

Is there any progress on this issue?
I also have several multi-output regressions to which I would like to apply auto-sklearn.

[mfeurer]

No, there's no one working on this. Do you want to contribute this feature?

[berisfu]

If the y label is (longitude,latitude),which means i wanna predict a location, auto-sklearn can support? I think there are numerous case about geo location，for example to predict where the user will drop off or pickup for Uber.

[mfeurer]

Auto-sklearn currently does not implement this feature. While we think that this feature would be good to have, I doubt that anyone from our team will implement this in the near future. Therefore, any contribution in getting this feature into Auto-sklearn would be greatly appreciated.

[Skylion007]

What would be needed to get this feature in? This library is missing crucial functionality without supporting these two problem types. I'd be willing to look into this if I can get an idea of how much effort it would take.

[mfeurer]

From the top of my head:

• making sure that the metrics work with this kind of data
• making sure that all the models are able to train on this kind of data
• making sure that ensembles work on this kind of data
  And then doing an integration check.

Point 2 is somewhat optional if only having unsupervised preprocessing and random forests for classification/regression are fine.

[Skylion007]

I mean theoretically every regressor can support it via this cludge? Of course, that's not optimal, but it will work since it cannot correlate data between output but it will give some functionality instead of just erroring. There is a multiclassification class that does the same thing.

For metrics, the following metrics are supported for multioutput according to the doc:

mean_squared_error, mean_absolute_error, explained_variance_score and r2_score.

Following classifiers support multilabel without the cludge.

It's difficult to find a list of regressors that support multioutput but it looks DecisionTrees and forms of Linear regressions along with their variants support it out of the box without the multioutput cludge.

That seems to be what I can find in the latest about which metrics and models work. Haven't found much about ensemble for multioutput regression but at least some of them support multiclass models. Given the list of regressors and models that support it, what would need to be done?

[Skylion007]

Alternatively, RegressorChain will be in the next version of Sklearn so that might be easier to work with: https://github.com/scikit-learn/scikit-learn/pull/9257/files

We already have ClassiferChain after all.

[mfeurer]

Yes, that's what I meant that wrapper which you'd need to plug around all kinds of classifiers - but I think that's secondary to get some basic functionality. Also, I don't think that ClassifierChain and RegressorChain are easily applicable in Auto-sklearn as it would be unclear how to tune their hyperparameters in a fast way.

Regarding the ensemble: Auto-sklearn uses an ensemble to post-hoc combine the models into an ensemble - ideally that one still works afterwards.

I think it would be easiest to start by adding multilabel regression (if that's of interest to you) by:

• adding a new flag handles_multilabel_regression to all models which support it in this directory: https://github.com/automl/auto-sklearn/tree/master/autosklearn/pipeline/components/regression
• add a new task type to the constants: https://github.com/automl/auto-sklearn/blob/master/autosklearn/constants.py
• check for that task type in this util file: https://github.com/automl/auto-sklearn/blob/master/autosklearn/util/pipeline.py
• Make the abstract regression component aware of multilabel regression, similar to the abstract classification component.
• Add a check for multilabel regression here as done for multiclass and multilabel
• Make the estimators and underlying base class aware of multilabel regression.

Please excuse that this is rather complicated and not all in one place, but we didn't desgin Auto-sklearn to be extensible for different tasks.

[charlesfu4]

My thesis topic is quite related to this issue, which is forecasting multi-output electricity load. I wonder if the team is working on multi-output auto regressor. If not, I will be willing to try that.

[charlesfu4]

The reason why we didn't implement anything together with the MultiOutputClassifier and MultiOutputRegressor is that they didn't nicely support partial_fit and models with warmstarts. As partial_fit seems to be added in sklearn==0.19 we can hopefully add this feature then.

Does it mean that the meta-learning part in your pipe-line will not be conducted well if I try to implement MultioutputClassifier directly on autosklearn? Since I read the paper that you published and knew that meta-learning makes use of warmstart.

[mfeurer]

Thanks to @charlesfu4 we now have multi-output regression available in the development branch, and it will be available in the next release of Auto-sklearn.

[mfeurer]

Closing this as scikit-learn still doesn't support multiclass-multioutput support. We can create a new, clean issue for this once scikit-learn provides metrics to evaluate multioutput-multiclass predictions.

[tron27]

Matthias,
It seems as though scikit-learn now offers/supports multiclass-multioutput support. See the link below:
https://scikit-learn.org/stable/modules/multiclass.html

I see that you mentioned almost two years ago that you may have a clean issue for this once scikit-learn provides metrics to evaluate multioutput-multiclass predictions. Do you know where you and your team is with this. As mentioned in some of the earlier messages, I would also like to apply automl to predict a location (latitude/longitude). Thanks for all that you do! It's greatly appreciated.

Eric

[eddiebergman]

Hi @tron27,

Can you make a new issue about this and I can add it to #1677

Best,
Eddie

[tron27]

Good morning Eddie,
Sure, I can make a new issue about this so that you can add it to #1677.

Thanks,
Eric

[tron27]

Good day Eddie,
I have created a new issue about this. You can find it here:

#1685

Thanks,
Eric