Improve PyPI package declared license detection

[pombredanne]

The goal of this ticket is to improve PyPI package license detection across the board. While scancode-toolkit's PyPI package detection is pretty good, there a few repeat cases where license information is not properly gathered from PyPI package metadata. Usually this is because a declared_license value contains things we did not expect (like a URL) or is improperly formed.

Resolving this would likely require a mix of:

• adding new license detection rules to scancode,
• adding new and improved code to handle the specific patterns of license,
• creating new license mappings
• and possibly working with upstream maintainers to improve their license declarations.

The approach should be to start with a complete data set of all package manifests and find patterns of license issues and establish the baseline, possibly with classifiers and ML. The end results should be a significant improvement to the license detection quality for the PyPI packages.

This https://github.com/pypa/bandersnatch/ and the PyPI API may help collect a list of all declared licenses.

See also https://www.python.org/dev/peps/pep-0639/ and may be #253 too

There are also other related ticket for other package types such as:

• Improve license detection of declared RPM licenses #2412 for RPM that has some detailed examples

And a project idea: https://github.com/nexB/aboutcode/wiki/Project-Ideas-Improve-PyPI-package-license-detection