Turing 1.0.0, AD: Where to put the testing functionality?

[penelopeysm]

Spin-off from #2411, please see that for more context

Introduction

As work towards Turing 1.0.0 we would like to have:

1. A series of test models that aim to cover various functionality used inside DPPL models
2. A few functions to test that they evaluate correctly with a given AD backend
3. Possibly functions to benchmark them

I did some very preliminary work sketching out what this might look like here: https://github.com/penelopeysm/ModelTests.jl

The question here is: where should this functionality live? The two major options are:

1. Keep them inside DynamicPPL itself, i.e. in some DynamicPPL.TestUtils module. The demo models are already there, it would then be a case of adding the testing functionality in there.
2. Move them to a separate Julia package, in a separate repo. This would essentially mean us building out the ModelTests repo and moving it under the TuringLang organisation.

[penelopeysm]

I'd actually like to suggest a third option, which is in-between these two:

3. Put them inside a DynamicPPLTest package, which also lives inside the DynamicPPL GitHub repository.

DynamicPPL would then be a dependency of DynamicPPLTest. Then, DynamicPPLTest would then be a test dependency of both DynamicPPL as well as other packages (like Turing, or any AD backend interested in testing against DPPL models).

This is mainly inspired by DifferentiationInterface, which has exactly the same setup: https://github.com/JuliaDiff/DifferentiationInterface.jl. It contains: (1) the main package DifferentiationInterface, which contains the actual functionality for performing AD; and (2) the test package DifferentiationInterfaceTest, which contains testing and benchmarking functionalities that build on the main package and allows users to test their own backends against the main package.

Notice that this exactly mirrors our situation here! This way, we get to avoid shoving everything inside DynamicPPL itself (which I'm not a fan of simply based on principle); and hopefully, by keeping both packages within the same repo, we can use CI/CD + other tools to synchronise feature development + releases.

That 'hopefully' is carrying some weight here though. I have never investigated how well the Julia ecosystem supports monorepos, and I've not yet studied the setup that DifferentiationInterface uses very carefully. But, if there isn't a serious technical reason against this, I strongly believe that this setup would be the cleanest, especially in the long term.

[mhauru]

This would be the most elegant thing, I agree. I'm afraid of how much pain this could turn into if it goes badly against the grain of what Julia tooling, like the registrator, expects. Maybe I'm wrong though, and it could support this with ease, I haven't looked into it at all.

[willtebbutt]

I'd be interested in seeing how this monorepo design plays out. SciML have really gone to town on this design, splitting out all of their solvers into individual packages in a monorepo -- there's an interesting post here about the issues they've been running into. I'd be interested to know what you both think of the issues it discuses -- they're clearly experiencing an extreme version of the issues that we would be likely to encounter, but it's an interesting read regardless.