Tern fails in license_expression

[vargenau]

Describe the bug

Tern fails with the following command:

tern report -f spdxtagvalue -i apache/airflow:2.3.0b1-python3.10 -o airflow-tern2.11.0.spdx

Error in terminal

Traceback (most recent call last):
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 777, in validate
    parsed_expression = self.parse(expression, strict=strict)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 539, in parse
    tokens = list(self.tokenize(
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 603, in tokenize
    for token in tokens:
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 996, in replace_with_subexpression_by_license_symbol
    for token_group in token_groups:
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 935, in build_token_groups_for_with_subexpression
    tokens = list(tokens)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 597, in <genexpr>
    tokens = (t for t in tokens if t.string and t.string.strip())
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 921, in build_symbols_from_unknown_tokens
    for symtok in build_token_with_symbol():
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 901, in build_token_with_symbol
    toksym = LicenseSymbol(string)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 1213, in __init__
    raise ExpressionError(
license_expression.ExpressionError: Invalid license key: the valid characters are: letters and numbers, underscore, dot, colon or hyphen signs and spaces: 'This file is free software; as a special exception the author gives'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vargenau/ternenv/bin/tern", line 8, in <module>
    sys.exit(main())
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/__main__.py", line 311, in main
    do_main(args)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/__main__.py", line 123, in do_main
    crun.execute_image(args)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/analyze/default/container/run.py", line 82, in execute_image
    report.report_out(args, full_image)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/report/report.py", line 86, in report_out
    report = generate_report(args, *images)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/report/report.py", line 41, in generate_report
    return generate_format(
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/report/report.py", line 56, in generate_format
    return mgr.driver.generate(images, print_inclusive)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/formats/spdx/spdxtagvalue/generator.py", line 141, in generate
    report += mhelpers.get_image_block(image_obj, template) + '\n'
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/formats/spdx/spdxtagvalue/image_helpers.py", line 134, in get_image_block
    pkg_block = get_image_packages_block(image_obj, template)
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/formats/spdx/spdxtagvalue/image_helpers.py", line 37, in get_image_packages_block
    block += phelpers.get_package_block(package,
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/formats/spdx/spdxtagvalue/package_helpers.py", line 105, in get_package_block
    block += 'PackageLicenseDeclared: ' + spdx_common.get_package_license_declared(
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/formats/spdx/spdx_common.py", line 61, in get_package_license_declared
    if is_spdx_license_expression(package_license_declared):
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/tern/formats/spdx/spdx_common.py", line 54, in is_spdx_license_expression
    return licensing.validate(license_data).errors == []
  File "/home/vargenau/ternenv/lib/python3.10/site-packages/license_expression/__init__.py", line 780, in validate
    expression_info.invalid_symbols.append(e.token_string)
AttributeError: 'ExpressionError' object has no attribute 'token_string'

Expected behavior
Tern should not fail.

Environment you are running Tern on
Enter all that apply

• Output of 'tern --version'

tern --version
Tern version 2.11.0
   python version = 3.10.7 (main, Nov 24 2022, 19:45:47) 

• Operating System: Ubuntu 22.10
• Python version: 3.10.7

[rnjudge]

Same issue as: #1199 -- I will create a workaround until the license_expression library can fix it (aboutcode-org/license-expression#76)

[pombredanne]

@rnjudge as mentioned in aboutcode-org/license-expression#76 and in aboutcode-org/license-expression#70 you should IMHO call scancode toolkit directly to get a proper license detection. The license expression library is not the license detection engine.

[rnjudge]

@pombredanne thanks for your input! Unfortunately, I don't know when I will have cycles to upgrade the license detection to Scancode and given that I can't even install Scancode on my M1 right now, I am hesitant to add it as a hard requirement for Tern users that also may be using M1s. Also, recent scancode updates appear to have broken Tern's parsing of scancode data so again hesitant to embed scancode in Tern when breaking changes cascade down to us. That being said, I do still want to look at what Scancode can offer here and see what we can do to improve license detection in Tern.

[timbrown5]

For anyone else still hitting this, it looks like this hasn't made it into a PyPi release.
As such, it is still and issue when using the Dockerfile from the repo https://github.com/tern-tools/tern/blob/main/docker/Dockerfile#L11

[rnjudge]

Hi @timbrown5 I am hoping to cut a release with this fix soon. I've been waiting on @pombredanne and the Scancode team to fix/offer a workaround for aboutcode-org/scancode-toolkit#3205 so I could also include a fix for #1202 but it may be a while for that. Stay tuned in the next few weeks for a patched release.