lightning vs pytorch-lightning

[adamjstewart]

I'm seeing conflicting documentation about the name of this library. It seems like the library was formerly called pytorch-lightning, and imported as such, but was later renamed. But both names still get updated on PyPI, and the documentation contains references to both. A few questions:

1. Are lightning and pytorch-lightning in fact identical, or does the former include more things than the latter?
2. Are there plans to deprecate pytorch-lightning in favor of lightning?
3. Which name is recommended for pip installs and imports?

[w2ex]

Someone please correct me if I am wrong, but as far as I know :

1. Pytorch-lightning became lightning with the 1.8.0 release. In the meantime, pytorch-lightning will be deprecated but is still working (for your imports, pip install and so on).
2. I think so but don't know when
3. Lightning being the new name it is the one being recommended

[adamjstewart]

Thanks for the clear and concise answer! We'll port TorchGeo from pytorch-lightning to lightning in that case. Should be only a few lines of code to change, we don't import in too many places.

[awaelchli]

+1 for @w2ex answer
Regarding the "when". We don't know! For now, we just wait until the "pytorch_lightning" package gets less popular, and then we do a proper deprecation and announcement. For now, we removed any deprecation messages in the docs, and continue to ship it :)) Meanwhile, all new docs, examples and materials that come out will promote the "lightning" package. Updating TorchGeo is a great idea, thanks for doing that!

[adamjstewart]

lightning seems to have significantly more dependencies than pytorch-lightning, leading me to believe that lightning and pytorch-lightning are not in fact identical

[awaelchli]

Yes it is true that it contains more dependencies. Because the lightning package looks like so:

lightning.pytorch (previously known as pytorch_lightning)
lightning.fabric (what we just launched called Fabric)
lightning.app (our cloud-based app framework for developing models and products)

The new dependencies that you see come from the inclusion of lightning.app. For example, this requires fastapi, arrow, click etc. which you didn't see with pytorch_lightning. I hope this clears it up a bit.

[adamjstewart]

I wonder if it would make sense to only install the dependencies for the component(s) you use. Something like:

$ pip install lightning[pytorch,app]

I realize that this is a step in the opposite direction since you just merged all of these together, but lightning now has quite a lot of dependencies...

[maxencealluin]

The naming is extremely confusing and inconsistent, reading their official announcement for 2.0 still raises questions: https://lightning.ai/pages/blog/introducing-lightning-2-0.
My understanding now is that there is a new library called Lightning Fabric, often called 'Lightning' that gives more control than regular Pytorch Lightning. Pytorch Lightning still exists and has been updated to 2.0.

[adamjstewart]

I think lightning-fabric is something different, no? https://pypi.org/project/lightning-fabric/

[w2ex]

Yes, Fabric is some kind of lightweight version of Lightning (it used to be referenced as lighning lite IIRC).

[awaelchli]

Hey, sorry about the confusion. About the lightning-fabric package on pypi: Yes, this is there but it is undocumented and unsupported. Because the imports for users won't work as intended and as shown in the docs. (I think we should not have published it at all). Please delete this from your memory 😄

Important to know going forward from 2.0, everything is grouped under the lightning package like so:

lightning.pytorch (previously known as pytorch_lightning)
lightning.fabric (what we just launched called Fabric)
lightning.app (our cloud-based app framework for developing models and products)

Our Lightning 2.0 launch is still ongoing and over the next few days, we will get the message out and educate people about these names and the brand going forward. I hope it is a little bit more clear now.

[dsuess]

Given that lightning lists pytorch-lightning as a dependency, how do these two packages interact? Should the versions of pytorch-lightning and lightning match? If so, can we put this as a constraint in the dependencies of the lightning package?

[adamjstewart]

If I'm understanding correctly, they don't interact at all and pytorch-lightning is only added "for backwards compatibility", see #17099. Personally, I would be in favor of removing this dependency so I can properly test that I no longer use pytorch-lightning anywhere.

[adamjstewart]

Was Lightning renamed back to PyTorch Lightning?

[adosar]

Weird but: pip install lightning installs also pytorch-lightning.

[adamjstewart]

Yep, looks like this was intentional for "backwards compatibility": #17099

I don't love this, wish things would be distributed under a single name so fewer packages need to be installed.