Custom Surrogate advice (starting with an EvalML pipeline with uncertainties from bootstrapped datasets)

[JanetMatsen]

Thank you for your work. This package is super cool and is being developed very rapidly!

I have a model I'd like to make a Surrogate for. The point-estimates for the posterior would come from an EvalML pipeline and the uncertainty values come from models of the same architecture that were trained on bootstrap resampled datasets.

I had something working in version 0.10.0, but am coming back to this and see a lot has changed as of 0.11.0.

I see you removed examples/Custom_Surrogates/custom_architecture_sklearn.py, which I mostly followed to get my example up. Now I see two choices:
(1) Try to use sklearn-onnx. I suspect this is a bad idea because while sometimes our model looks like sklearn there's a lot more going on.
(2) Read the examples in baybe/surrogates and make our own class that inherits from baybe/surrogates/base.py.

I'm moving toward (2) but wanted to thank you and see if this is what you'd recommend.

Thank you,
Janet

[Scienfitz]

@JanetMatsen thanks for the kind words and for the interest

Indeed some of the examples have been removed, mainly due to two reasons

• Our surrogate structure was refactored quite a bit, essentially making the existing examples useless, didnt have the time to translate the examples so they were removed for safety
• We realized some problems like eg models that do not provide a joint posterior cannot do proper batch recommendation

This was mostly driven by @AdrianSosic and Im sure he will comment with more details.

However, I believe, independent of all details, the best path for you could simply be to implement your model as you say derived from our Surrogate base. In particular, if you have a model ensemble like the bootstrapped one you mention, then you could take a close look at our random forest implementation. It uses the single predictors in the ensemble and botorch.models.ensemble.EnsemblePosterior to being able to provide a joint posterior (if I recall correctly).

Seems to me the most straight forward template for you. Additional considerations for scaling might apply to your model (the random forest doesnt scale as its not needed, but for how to do it in our framework you could look at other models or follow up via discussion)

Perhaps batch recommendation is also irrelevant to you, then pretty much any non-GP model implementation could be followed.

[AdrianSosic]

Hi @JanetMatsen, many thanks also from my side 😃 Agree with everything @Scienfitz wrote and also suggest using the EnsemblePosterior in your case, just as we did with the random forest.

Here some more context for better understanding what was the reason for the change:

• Historically, our first surrogate layout was there even before we really considered situations where batch recommendations where required. While the setup was not "wrong", it never really supported proper batch optimization. That is, while the old code would give you several recommendations if you requested them, points returned in a batch by models other than the GP would not be optimized jointly, i.e. no diversification mechanism was in place.
• The reason for this was that there was a hard-coded isotropic Gaussian assumption in place, meaning the models would not be able to propagate any covariance information. However, this is crucially necessary for proper batching.
• The recent refactoring introduced some major changes, such that the surrogates now actually return a Posterior object, which is capable of capturing that covariance information.
• If you have a model where the prediction is made using several different predictions, the EnsemblePosterior wrapping these predictions can represent the necessary covariance structure for you.

Implementation-wise, you can:

• Either inherit from our Surrogate base class. This comes with the advantage that input/output scaling is automatically done for you and fully customizable but has the "drawback" that it involves subtyping and requires you to understand the logic of that class.
• Or just write your own class that conforms with our SurrogateProtocol. This gives you full control and minimal requirements (i.e. you only need to implement two simple methods: one for fitting the model to data according to your custom logic and one for exporting that logic to botorch in the form of a Model) but has the disadvantage that you need to take care of everything yourself.

If you need further assistance, don't hesitate to ask. If you enjoy the package, consider leaving a star ⭐ 🙃

[JanetMatsen]

Hi! This is working great for me. I am playing on comparing GPs, Random, and AutoML surrogates with a synthetic fitness landscape.

And yes, I starred your repo. Nice work!

[AdrianSosic]

Glad to hear! Closing the discussion now but feel free to reopen at any time in case of further questions 👍🏼

[Scienfitz]

@JanetMatsen sounds interesting, in case the results can be shared publicly I'd be interested to see them!