Looking for feedback on an idea: Inheritance hierarchy of LightningDataModules

[lebrice]

Hi all! I'm Fabrice. I'm doing a masters in ML, my main focus is Continual (Supervised & Reinforcement) Learning.
@Borda suggested I move this post over from slack to GitHub, so that we can hopefully have a structured discussion. Here it is:

I'd like your opinion on an idea of mine (which is also at the core Sequoia, an open-source package i'm buillding, which is based around Pytorch-Lightning):

Here's the gist of it:

1. Take the LightningDataModule class, and add a new method to it: apply(Method) -> Results . Call this new subclass a Setting.
2. Organise these Settings (machine learning research problems) into a tree-shaped inheritance hierarchy based on their different assumptions, where the most "general" settings are placed on top, and the most specific settings are at the bottom. (see the image below)
3. When creating an algorithms (Methods), they specify which Setting they target, and are reusable onto any Setting below it through inheritance.

Here is why I think this is would be useful to the research community:

• Methods can be reused across settings making it easy to conduct vast experiments! --> Easier code reuse!
• When new Settings are introduced, they can already use any Method applicable to their parent! --> "Free" baselines for new settings
• This "tree of research" grows as new methods and settings are added, giving the methods more settings to tackle, and the settings more methods to test out!
• This also gives an incentive for everyone to try and tackle more and more general/challenging research problems.

I'm currently working to apply this idea to organize the plethora of settings that exist in Continual Supervised Learning and Continual Reinforcement Learning. I'd love to hear your thoughts/comments.

[carmocca]

Organise these Settings (machine learning research problems) into a tree-shaped inheritance hierarchy based on their different assumptions, where the most "general" settings are placed on top, and the most specific settings are at the bottom. (see the image below)
When creating an algorithms (Methods), they specify which Setting they target, and are reusable onto any Setting below it through inheritance.

This is more-or-less the pattern followed in Flash
where, for example, an ImageClassifier subclasses ClassificationTask which subclasses Task which subclasses LightningModule

Although, currently, there is no mechanism to explicitly define which Setting (aka Task in flash) are targeted for each algorithm.

cc @ethanwharris

[lebrice]

Hey @carmocca, thanks for your reply!

Interesting! I did take a brief look at Flash before, there are definitely things in common, but would you mind clarifying a few things for me?

(By the way, I'll use "settings" --> research problems (i.e. LightningDataModule subclass), and "method" --> algorithm, perhaps a LightningModule subclass)

• Is there a distinction between what I'd call "settings" and "methods" in Flash? Do LightningDataModule's play the role of the Setting? or does the LightningModule/Task?
• You seem to have the inheritance hierarchy setup on the "method" side, rather than on the "setting" side, correct? If so, then could you give me an example of how polymorphism would work on your end?

Thanks again!

[tchaton]

Dear @lebrice,

Thanks for reaching out and opening this interesting discussion. I went through Sequoia codebase to better understand where you are coming from and I wanted to give you a shot out ! Definitely great work there :)

Concerning Flash

Here is the high level diagram of Flash current API.

After several iteration, we decided to have what you call Setting / Method merged as one component which is called a TrainingStrategy available to the Task though a registry.

ImageClassifier(backbone="resnet50", training_strategies=["meta_learning-metaoptnet", "ssl-simclr"])

Under the hook, the training_strategies can access the Preprocess / DataModules datasets and modify them as needed to adapt them to the Method. By doing so, there is the separation of concerns between how data get loaded, preprocessed and how they get used by the Method.

Note: The TrainingStrategies API is still in developement and likely to change.

Concerning the discussion

I don't believe Lightning should offer any opinionated solution such as Flash or Sequoia ones.
Lightning is converging to minimal framework as PyTorch, so users are free to create the structure they need.
Lightning shouldn't be more than organised / simplified PyTorch

However, there is definitely some good synergy between Flash and Sequoia which might be worth to investigate if you are willing to.

Best,
T.C