Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to scan repository with FossID: Cannot deserialize value of type java.util.LinkedHashMap<java.lang.String,java.lang.String> #8462

Open
realbigflo opened this issue Mar 25, 2024 · 8 comments · May be fixed by #8541
Assignees
Labels
bug Issues that are considered to be bugs scanner About the scanner tool

Comments

@realbigflo
Copy link

When trying to create a scan with FossID (workbench version: 2023.3.0) ORT scan step fails.
I tried different internal repositories all failing with the same pattern. Is there anything I can provide as debug more than the traceback? Is there any option to increase the debug output even further to get the details of the API calls that are made towards FossID (assuming, that there is a mismatch between what ORT gets and what is expected)?

16:31:44.135 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - Default branch is 'develop'.
16:31:44.135 [main] WARN  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - No project revision has been given.
16:31:44.136 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossIdUrlProvider - URL mapping applied to <https git url>: Mapped to <ssh git url>.
16:31:44.137 [main] WARN  java.util.List - No recent scan found for project revision null. Falling back to default branch scans.
16:31:44.137 [main] WARN  java.util.List - No recent default branch scan found. Falling back to old behavior.
16:31:44.137 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - Found 0 scans.
16:31:44.137 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - No scan found for <ssh git url> and revision <revision>. Creating origin scan...
16:31:44.137 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossIdNamingProvider - Parameterizing the name with pattern '$Var1_$Var3'.
16:31:44.138 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - Creating scan 'jenkins_library_test'...
Exception in thread "main" com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.util.LinkedHashMap<java.lang.String,java.lang.String>` from Array value (token `JsonToken.START_ARRAY`)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 49] (through reference chain: org.ossreviewtoolkit.clients.fossid.EntityResponseBody["data"])
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1767)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1541)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromArray(StdDeserializer.java:222)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:457)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:440)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1493)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:348)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
        at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
        at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2125)
        at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1501)
        at retrofit2.converter.jackson.JacksonResponseBodyConverter.convert(JacksonResponseBodyConverter.java:33)
        at retrofit2.converter.jackson.JacksonResponseBodyConverter.convert(JacksonResponseBodyConverter.java:23)
        at retrofit2.OkHttpCall.parseResponse(OkHttpCall.java:246)
        at retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:156)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
@sschuberth
Copy link
Member

@nnobelis could you please look into this?

@sschuberth sschuberth added bug Issues that are considered to be bugs scanner About the scanner tool labels Mar 25, 2024
@sschuberth
Copy link
Member

@alexandruz any insights what might have changed in FOSSID'S data model?

@nnobelis
Copy link
Member

nnobelis commented Apr 8, 2024

@realbigflo Unfortunately, we cannot log the requests to FossID as they contain credentials.

We are also running this version and we don't get the the error.
Looking at your stack trace it seems FossID returns now for service.createScan an array and not a map anymore, but ORT should be able to handle this case.

If you want, reach me internally and we can look at this problem together.

@realbigflo
Copy link
Author

While debugging with @nnobelis we figured out, the problem is caused by an access problem of Fossid for the specific git repository. I thought initially, that the ssh key was added correctly to the bitbucket instance but unfortunately this wasn't the case.

This is the request that should be send to the FossID instance:

curl --header "Content-Type: application/json" --request POST --data "{\"action\":\"create\",\"group\":\"scans\",\"data\":{\"username\":\"username\",\"key\":\"mykey\",\"project_code\":\"test_project\",\"scan_code\":\"test_ssh_cred_fail\",\"scan_name\":\"test_ssh_cred_fail\",\"git_repo_url\":\"ssh git repo url\",\"git_branch\":\"develop\"}}" <fossid REST endpoint>
{"operation":"scans_create",
 "status":"0",
 "data":[{"code": "RequestData.Base.issue_with_executing_command",
              "message":"Field git_repo_url: there was an issue executing command: timeout 200 git ls-remote 'ssh git repo' 2>&1. Exit status: 128. Output: Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists.",
              "message_parameters":{"fieldname":"git_repo_url",
                                                    "cmd":"timeout 200 git ls-remote 'ssh git repo' 2>&1",
                                                    "exitStatus":128,
                                                    "out":"Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists."}
              }],
"error":"RequestData.Base.issues_while_parsing_request",
"message":"These issues were found while parsing the request:",
"message_parameters":[]
}

After correcting the authentication issue, the result of the request looks like:

{"operation":"scans_create",
 "status":"1",
 "data":{"scan_id":23732},
 "message":"Scan test_ssh_cred_fail has been created."
  }

The data field in the failure case is an array with a single element (map) while in the good case, it is only a map.

@nnobelis
Copy link
Member

nnobelis commented Apr 8, 2024

Thanks a lot @realbigflo : Yeah, it seems they changed the syntax of the error response, with even more polymorphism :(
I will have a look.

@nnobelis
Copy link
Member

nnobelis commented Apr 8, 2024

@sschuberth Could you maybe assign the issue to me ?

nnobelis added a commit to bosch-io/oss-review-toolkit that referenced this issue Apr 18, 2024
Under some rare circumstances, `createScan` can return an error message as
data payload instead of the scan id. This commit changes the function's
signature to make it polymorphic.

Fixes oss-review-toolkit#8462.

Signed-off-by: Nicolas Nobelis <[email protected]>
@nnobelis
Copy link
Member

@sschuberth I started to work on it here: #8541

Unfortunately, the exception is still present: the polymorphic deserializer currently allows to process multiple responses for the same function: T, [T], { "xxx", T }.

Unfortunately, in this case, the responses are either data: { "scan_id": N } or something totally different e.g.

"data":[{"code": "RequestData.Base.issue_with_executing_command",
              "message":"Field git_repo_url: there was an issue executing command: timeout 200 git ls-remote 'ssh git repo' 2>&1. Exit status: 128. Output: Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists.",
              "message_parameters":{"fieldname":"git_repo_url",
                                                    "cmd":"timeout 200 git ls-remote 'ssh git repo' 2>&1",
                                                    "exitStatus":128,
                                                    "out":"Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists."}
              }],

I still need to work this out.

@nnobelis
Copy link
Member

As I don't see how to fix this without an ugly hack, I asked our OPS to create a ticket at FossID support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issues that are considered to be bugs scanner About the scanner tool
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants