[BUG] jarsigner + jca reports that entries in certificate chain are invalid

[3millionminds]

Describe the bug

When using a valid, DigiCert issued, non-exportable Azure Key Vault certificate to sign a .jar file with jarsigner + jca 2.8.2 OR jarsigner + jca 2.9.0-beta.3, the file is signed with apparently no issues.

>>> Signer
    X.509, CN=██████████ Limited, O=██████████ Limited, L=London, C=GB
    [trusted certificate]

jar signed.

However, upon verifying the file, a warning is produced:

This jar contains entries whose certificate chain is invalid.
Reason:
    PKIX path building failed:
        sun.security.provider.certpath.SunCertPathBuilderException:
            unable to find valid certification path to requested target

This prevents the jar file to run on high-security systems, and we are unable to run the applications.

Exception or Stack Trace

jca282_signing.log
jca282_verify.log
jca282_environment.log

jca290_signing.log
jca290_verify.log
jca290_environment.log

To Reproduce

Steps to reproduce the behavior:

1. Sign a JAR file with jarsigner and azure-security-keyvault-jca-2.8.2.jar (or 2.9.0-beta.3), with a valid certificate issued by DigiCert and stored in AzKV.

   $ jarsigner -keystore NONE -storetype AzureKeyVault \
           -signedjar SIGNED_SignMe.jar SignMe.jar "██████████CodeSigning" \
           -verbose -storepass "" \
           -tsa "http://timestamp.digicert.com/" \
           -providerName AzureKeyVault \
           -providerClass com.azure.security.keyvault.jca.KeyVaultJcaProvider \
           -J-Dazure.keyvault.uri="██████████" \
           -J-Dazure.keyvault.tenant-id="██████████" \
           -J-Dazure.keyvault.client-id="██████████" \
           -J-Dazure.keyvault.client-secret="██████████" 
   [lines omitted]
   >>> Signer
       X.509, CN=██████████ Limited, O=██████████ Limited, L=London, C=GB
       [trusted certificate]
   
   jar signed.
   

2. Verify it's signature

       $ jarsigner -verify SIGNED_SignMe.jar 
   
       jar verified.
   
       Warning: 
       This jar contains entries whose certificate chain is invalid.
       Reason:
           PKIX path building failed:
               sun.security.provider.certpath.SunCertPathBuilderException:
                   unable to find valid certification path to requested target
   
   

Expected behavior
There .jar should be signed, and there should be no warnings when verifying the signature.

Screenshots

Setup (please complete the following information):

• OS: (containerized) Red Hat Enterprise Linux release 8.10 (Ootpa)
• IDE: N/A
• Library/Libraries:
  • com.azure:azure-security-keyvault-jca:2.8.2
  • com.azure:azure-security-keyvault-jca:2.9.0-beta.3 (built from rev ac9ee6330b6)
• Java version: openjdk-1.8.0_422
• App Server/Environment: N/A
• Frameworks: N/A

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[vcolin7]

HI @3millionminds, thank you for filing this issue, I'll pass it along to the team.

@saragluna, @rujche Could you take a look when you get a chance? Thanks!

[rujche]

Hi, @3millionminds

Thanks for reaching out.

It's weird that when you sign the jar, the signer only has one cert, and the cert is trusted

In my understanding, it should be a chain, and only the root cert is trusted. Here is an example: #41303 (comment)

Could you please add verbose to the verify command and share the output log? Here is an example of adding verbose:

jarsigner -verify -verbose -certs SIGNED_SignMe..jar

Caution: remove private information when upload the log file, just like you did above.

[3millionminds]

Hello @rujche, thank you for picking up this issue.

Here are the logs:
jca282_verbose_verify.log
jca290_verbose_verify.log

I will try and import the root ca into the local keystore first and then follow the signing&verification process.
The -keystore NONE makes it a bit misleading since the assumption is that it is not using the local keystore at all.

I will return with the results.

[3millionminds]

@rujche if import the DigiCert CA into the local keystore it works on the machine where it is imported in the keystore.
If I verify on a different machine, same thing as before happens.

    Warning: 
    This jar contains entries whose certificate chain is invalid.
    Reason:
        PKIX path building failed:
            sun.security.provider.certpath.SunCertPathBuilderException:
                unable to find valid certification path to requested target

Same thing if I import only our certificate.

[rujche]

Hi, @3millionminds

1. About importing the DigiCert CA into the local keystore

if import the DigiCert CA into the local keystore it works on the machine where it is imported in the keystore.
If I verify on a different machine, same thing as before happens.

By saying import the DigiCert CA into the local keystore, do you mean adding to cacerts like this:

keytool -import -alias digicertca -keystore $JAVA_HOME/jre/lib/security/cacerts -file DigiCertCA.crt

2. Root CA cert should be added into cacerts

To verify a jar signed by a cert, the cert's root CA cert should be added in to $JAVA_HOME/jre/lib/security/cacerts. In your case, it's DigiCert CA. In some cases, the DigiCert CA was already added into cacerts when install Java. If cacerts not include DigiCert CA then you should add it manually. Refs: https://knowledge.digicert.com/general-information/importing-digicert-pki-ca-certificates-into-java-ca-certificates-keystore

3. Seems there is only one cert in the cert chain.

As mentioned above #41832 (comment), it's weird that in your provided log, the signer only has one cert, is this cert self-signed? it shouldn't be. It should be signed by DigiCert CA. Here is the screenshot of your provided log:

In my test case (#41303 (comment)), both sign and verify showed 3 certs in the cert chain. Here are the screenshots of my test case:

[3millionminds]

Hi @rujche

1. Yes, that is exactly how I import it.
2. Yes, I have imported the DigiCert CA into cacerts.
3. The certificate is not self-signed, it is signed by DigiCert themselves.

After installing the DigiCert CA on the machine where I do the signing, I can verify the jar file, it no longer complains about the certificate chain.

But, as mentioned here, when trying to verify the jar on a different machine, the issue returns. I imagine if they had the certificate in their cacerts it would also work for them.

Our usecase is the following: the jars are built, signed, and then packaged into a larger application, which is delivered to the end clients. This software is then deployed on hundreds of computers where, due to security restrictions, it cannot run.

Before switching to Azure Key Vault, a local PFX certificate was used for signing.
Signing was done by Maven during the build, by leveraging the maven-jarsigner-plugin.
I have asked around and nobody is aware of us delivering any certificates, or keystores, or anything else except the packaged software itself.
However, when using this method, the packages could be verified and run on these client systems.

[rujche]

Hi, @3millionminds

1. About the cert chain

3. The certificate is not self-signed, it is signed by DigiCert themselves.

In my understanding, if it is signed by DigiCert, then the cert chain should have more than one cert. When sign and verify jar file using the cert, it should display all the certs in the cert chain. But your log only show one cert. I'm confused about this.

2. About your use case.

Before switching to Azure Key Vault, a local PFX certificate was used for signing.
Signing was done by Maven during the build, by leveraging the maven-jarsigner-plugin.
I have asked around and nobody is aware of us delivering any certificates, or keystores, or anything else except the packaged software itself.
However, when using this method, the packages could be verified and run on these client systems.

Now I understand your case. Now I'm also confused why it work with the local pfx file but not work with Azure Key Vault method.

3. How to do next step?

I suggest you do these things:

1. Show the cert chain by method like this: https://serverfault.com/questions/1011294/how-to-view-certificate-chain-using-openssl The cert chain should have more than one cert. If the cert chain only has one cert, investigate why it only has one cert.
2. Sign the jar file by both local method and Key Vault method, and confirm the cert chain is the same with step 1. The cert chain should have more than one cert. If the cert chain only has one cert, investigate why it only has one cert.
3. Verify the signed jar, and confirm the cert chain is the same with step 1. The cert chain should have more than one cert. If the cert chain only has one cert, investigate why it only has one cert.

If the problem still can not be solved after above steps, please share related log, and let's think about how to do next step.

[3millionminds]

Hello @rujche

1. Here is the certificate chain:

2. Unfortunately I can no longer sign it locally, since I don't have the PFX keystore. It is not provided by DigiCert. All I can get from them are either (a) the certificate chain, or (b) each individual certificate, in various formats, none of them usable by jarsiger (.pem, .crt, .cer, .p7b).
   If I try to import the .cer file:

   [root@5212ca8d81ab ~]# keytool -import -alias TestAlias -keystore $JAVA_HOME/jre/lib/security/cacerts -file chain.cer
   
   Enter keystore password:  
   keytool error: java.lang.Exception: Input not an X.509 certificate
   

   The .pem and .crt I can import but I cannot sign with:

   [root@5212ca8d81ab ~]# jarsigner -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -signedjar test.jar SignMe.jar TestAlias
   
   jarsigner: Certificate chain not found for: TestAlias.  TestAliasmust reference a valid KeyStore key entry containing a private key and corresponding public key certificate chain.
   

[rujche]

Hi, @3millionminds

1. About DigiCert root CA.

1. Here is the certificate chain:

In your screenshot, I see the root cert name (CN=digicert_trusted_root_g4).
The root cert seems already included in the cacerts. I confirmed by this command:

 keytool -list -keystore "C:\Program Files\Microsoft\jdk-17.0.11.9-hotspot\lib\security\cacerts"

Here is the output:

...
cn=digicert_trusted_root_g4,ou=www.digicert.com,o=digicert_inc,c=us, 13 Mar 2024, trustedCertEntry,
Certificate fingerprint (SHA-256): 55:2F:7B:DC:F1:A7:AF:9E:6C:E6:72:01:7F:4F:12:AB:F7:72:40:C7:8E:76:1A:C2:03:D1:D9:D2:0A:C8:99:88
...

Could you please confirm that the DigiCert root cert already included in your cacerts too?

2. Sign by the cert provided by DigiCert.

Unfortunately I can no longer sign it locally, since I don't have the PFX keystore. It is not provided by DigiCert.

I found this doc: https://knowledge.digicert.com/tutorials/export-your-ssl-certificate-using-the-digicert-certificate-utility-pfx-format

To sign a jar file, a private key is necessary. For security reason, sometimes the private key is not exportable. We can ignore the part of sign it locally for now.

3. What to do next step?

Please confirm these things:

1. Confirm the DigiCert root cert included in the cacerts in your environment.
2. If the root cert included in cacerts by default, investigate why import manually is necessary as you said above:

if import the DigiCert CA into the local keystore it works on the machine where it is imported in the keystore.
If I verify on a different machine, same thing as before happens.

[3millionminds]

Hello @rujche

Sorry for the delay, we had to stop debugging this issue for a while in order to manually sign, with a hardware token, a product release.

So, I have checked that the DigiCert root CA is included in the cacerts.
The output is below. There seem to be quite a few digicert related certificates there.
DigiCertCA is the alias of the one imported by me.

digicertassuredidrootca, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 3E:90:99:B5:01:5E:8F:48:6C:00:BC:EA:9D:11:1E:E7:21:FA:BA:35:5A:89:BC:F1:DF:69:56:1E:3D:C6:32:5C
digicertassuredidrootg2, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 7D:05:EB:B6:82:33:9F:8C:94:51:EE:09:4E:EB:FE:FA:79:53:A1:14:ED:B2:F4:49:49:45:2F:AB:7D:2F:C1:85
digicertassuredidrootg3, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 7E:37:CB:8B:4C:47:09:0C:AB:36:55:1B:A6:F4:5D:B8:40:68:0F:BA:16:6A:95:2D:B1:00:71:7F:43:05:3F:C2
digicertca, Sep 27, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 46:01:1E:DE:1C:14:7E:B2:BC:73:1A:53:9B:7C:04:7B:7E:E9:3E:48:B9:D3:C3:BA:71:0C:E1:32:BB:DF:AC:6B
digicertglobalrootca, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 43:48:A0:E9:44:4C:78:CB:26:5E:05:8D:5E:89:44:B4:D8:4F:96:62:BD:26:DB:25:7F:89:34:A4:43:C7:01:61
digicertglobalrootg2, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): CB:3C:CB:B7:60:31:E5:E0:13:8F:8D:D3:9A:23:F9:DE:47:FF:C3:5E:43:C1:14:4C:EA:27:D4:6A:5A:B1:CB:5F
digicertglobalrootg3, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 31:AD:66:48:F8:10:41:38:C7:38:F3:9E:A4:32:01:33:39:3E:3A:18:CC:02:29:6E:F9:7C:2A:C9:EF:67:31:D0
digicerthighassuranceevrootca, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 74:31:E5:F4:C3:C1:CE:46:90:77:4F:0B:61:E0:54:40:88:3B:A9:A0:1E:D0:0B:A6:AB:D7:80:6E:D3:B1:18:CF
digicerttlseccp384rootg5, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 01:8E:13:F0:77:25:32:CF:80:9B:D1:B1:72:81:86:72:83:FC:48:C6:E1:3B:E9:C6:98:12:85:4A:49:0C:1B:05
digicerttlsrsa4096rootg5, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 37:1A:00:DC:05:33:B3:72:1A:7E:EB:40:E8:41:9E:70:79:9D:2B:0A:0F:2C:1D:80:69:31:65:F7:CE:C4:AD:75
digicerttrustedrootg4, Sep 10, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): 55:2F:7B:DC:F1:A7:AF:9E:6C:E6:72:01:7F:4F:12:AB:F7:72:40:C7:8E:76:1A:C2:03:D1:D9:D2:0A:C8:99:88

For your second point, "investigate why import manually is necessary", I can relate the following:

1. When we sign .exe/.dll files with AzureSignTool, they get signed with the entire chain of certificates. We don't do anything else on the signing machines, or the "end-user" machines. We don't import any certificates, nothing. We just sign it, and the signature is valid.
   See here:
   

2. When we sign .jar files, with jarsigner and a DigiCert Hardware eToken, the entire certificate chain appears when verifying the signatures of the classes:

[entry was signed on 24.09.2024, 14:16]
>>> Signer
X.509, CN=██████████ Limited, O=██████████ Limited, L=London, C=GB
Signature algorithm: SHA256withRSA, 4096-bit key
[certificate is valid from 18.09.2024, 03:00 to 05.04.2027, 02:59]
C=US
Signature algorithm: SHA384withRSA, 4096-bit key
[certificate is valid from 29.04.2021, 03:00 to 29.04.2036, 02:59]
X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
Signature algorithm: SHA384withRSA, 4096-bit key
[trusted certificate]
>>> TSA
X.509, CN=DigiCert Timestamp 2023, O="DigiCert, Inc.", C=US
Signature algorithm: SHA256withRSA, 4096-bit key
[certificate is valid from 14.07.2023, 03:00 to 14.10.2034, 02:59]
X.509, CN=DigiCert Trusted G4 RSA4096 SHA256 TimeStamping CA, O="DigiCert, Inc.", C=US
Signature algorithm: SHA256withRSA, 4096-bit key
[certificate is valid from 23.03.2022, 02:00 to 23.03.2037, 01:59]
X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
Signature algorithm: SHA384withRSA, 4096-bit key
[certificate is valid from 01.08.2022, 03:00 to 10.11.2031, 01:59]

When signing the .jar with this hardware token, we are also able to verify the signatures on any other devices, without having to install any certificates on the systems.
I also did not install any other extra certificates on my local machine in order to sign. Not the DigiCert CA, not even our certificate.

So my guess is there is something not working properly in the way the certificates are retrieved from Azure KeyVault.
I don't understand, otherwise, how signing .exe/.dll with the same KeyVault but with a different tool works, and signing .jar with the same certificates from a local hardware token also works, but not from KV.

[dacron]

We have also hit this with a non-exportable HSM backed key. We have reissued the certificate multiple times and merged bundles in to new versions of KV certificates:

1. leaf
2. leaf -> intermediate
3. leaf -> intermediate -> root.

With just the leaf we can sign but it is not trusted until we add the intermediate in to the cacerts store. With scenarios 2 and 3 jarsigner reports that everything is signed successfully, but verify says otherwise:

WARNING: Signature is either not parsable or not verifiable, and the jar will be treated as unsigned. For more information, re-run jarsigner with debug enabled (-J-Djava.security.debug=jar).

With the debug enabled:

jar: beginEntry META-INF/MANIFEST.MF
jar: beginEntry META-INF/2024OVCS.SF
jar: processEntry: processing block
jar: beginEntry META-INF/2024OVCS.RSA
jar: processEntry: processing block
jar: Unsupported signer attribute: 1.2.840.113549.1.9.16.2.47
jar:
jar: Detected signature timestamp (#00:dc:50:89:c6:24:67:a6:ed:80:1e:ac:ca:90:98:99:45) generated on Fri Sep 27 16:14:05 BST 2024
jar:
jar: processEntry caught: java.security.SignatureException: Bad signature length: got 384 but was expecting 512
jar: done with meta!
jar: nothing to verify!
jar: Unsupported signer attribute: 1.2.840.113549.1.9.16.2.47

Our intermediate CA is DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1 and the root is DigiCert Trusted Root G4

[rujche]

Hi, @3millionminds

Sorry for late response.

When we sign .jar files, with jarsigner and a DigiCert Hardware eToken, the entire certificate chain appears when verifying the signatures of the classes:

Signer
X.509, CN=██████████ Limited, O=██████████ Limited, L=London, C=GB
Signature algorithm: SHA256withRSA, 4096-bit key
[certificate is valid from 18.09.2024, 03:00 to 05.04.2027, 02:59]
C=US
Signature algorithm: SHA384withRSA, 4096-bit key
[certificate is valid from 29.04.2021, 03:00 to 29.04.2036, 02:59]
X.509, CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US
Signature algorithm: SHA384withRSA, 4096-bit key
[trusted certificate]

This looks good because the whole chain appeared when sign the file.

What to do next

1. Now I guess the problem only happens when all the 3 condition happened:
   1.1. The cert chain has more the one cert.
   1.2. The private key is non-exportable.
   1.3. It is a HSM (hardware security module) backed key.
   Not I haven't confirm this yet. For the case that only satisfy 1.1 and 1.2, I already added unit test here: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/keyvault/azure-security-keyvault-jca/src/test/resources/certificate-util/SecretBundle.value

2. Out team is in Shanghai office, now we are on holiday until October 8. If you are available, could you please help to confirm this: This problem only happens when all the 3 above condition satisfied.

[saragluna]

Closing this issue now, please reopen it if the issue is not resolved.

[dacron]

@3millionminds did this work for you?

[dacron]

@saragluna this still doesn't work, please could you reopen?

If you use a standard RSA key it works with a self-signed multi-tier openssl-based certificate authority. The PEM file containing the signed certificate contains "root => intermediate => signed_cert" in that order. Verify works here if my root is in the cacerts.

If you use a key of type RSA-HSM and a DigiCert signed certificate it does not work. It doesn't work if I self sign a RSA-HSM backed cert either via the previously mentioned openssl-based CA. The verify of the DigiCert RSA-HSM cert shows:

[removed]@[removed] AzureKeyVault % jarsigner -verify -certs -verbose helloworld.signed.jar -J-Djava.security.debug=jar
jar: beginEntry META-INF/MANIFEST.MF
jar: beginEntry META-INF/2024OVCS.SF
jar: processEntry: processing block
jar: beginEntry META-INF/2024OVCS.RSA
jar: processEntry: processing block
jar: Unsupported signer attribute: 1.2.840.113549.1.9.16.2.47
jar:
jar: Detected signature timestamp (#00:94:53:7d:3d:f7:94:0a:c2:71:9f:ed:51:78:bd:b5:0e) generated on Tue Oct 29 16:13:50 GMT 2024
jar:
jar: processEntry caught: java.security.SignatureException: Bad signature length: got 384 but was expecting 512
jar: done with meta!
jar: nothing to verify!

         166 Tue Oct 29 16:13:48 GMT 2024 META-INF/MANIFEST.MF
         299 Tue Oct 29 16:13:50 GMT 2024 META-INF/2024OVCS.SF
       11666 Tue Oct 29 16:13:50 GMT 2024 META-INF/2024OVCS.RSA
           0 Wed Jan 15 14:43:32 GMT 2020 META-INF/
 m  ?    446 Wed Jan 15 14:21:40 GMT 2020 HelloWorld.class

  s = signature was verified 
  m = entry is listed in manifest
  k = at least one certificate was found in keystore
  ? = unsigned entry

jar: Unsupported signer attribute: 1.2.840.113549.1.9.16.2.47
- Signed by "CN=DigiCert Trusted Root G4, OU=www.digicert.com, O=DigiCert Inc, C=US"
    Digest algorithm: SHA256
    Signature algorithm: SHA256withRSA, 4096-bit key
  Timestamped by "CN=DigiCert Timestamp 2024, O=DigiCert, C=US" on Tue Oct 29 16:13:50 UTC 2024
    Timestamp digest algorithm: SHA-384
    Timestamp signature algorithm: SHA384withRSA, 4096-bit key

WARNING: Signature is either not parsable or not verifiable, and the jar will be treated as unsigned. For more information, re-run jarsigner with debug enabled (-J-Djava.security.debug=jar).

The verify is reporting that the signature is from the DigiCert root, and not the DigiCert issued code signing certificate. Interestingly enough if I run the META-INF/2024OVCS.RSA through openssl you can see that the full chain is there:

[removed]@[removed] AzureKeyVault % openssl pkcs7 -in extracted/META-INF/2024OVCS.RSA -inform DER -print | grep subject: -A6
          subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Trusted Root G4
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 02 0a 02 82 02 01-00 bf e6 90 73 68   0...........sh
--
          subject: C=US, O=DigiCert, Inc., CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 02 0a 02 82 02 01-00 d5 b4 2f 42 d0   0........../B.
--
          subject: [REMOVED]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 01 8a 02 82 01 81-00 9f ba 3f 95 15   0..........?..

The DigiCert root is also in the Java's cacerts.

The problem definitely is around HSM backed/non-exportable keys.

[3millionminds]

@3millionminds did this work for you?

Hello,

Unfortunately we have been unable to test.
We are unable to merge new certificates on top of the ones we already have in Azure Key Vault.

We have opened a MS ticket to see what options we have here.

[dacron]

More data points. Given an exportable RSA key type with the self-signed setup the output of verify is:

[removed]@[removed] AzureKeyVault % jarsigner -verify helloworld.signed.jar -verbose

s        166 Wed Oct 30 10:52:08 GMT 2024 META-INF/MANIFEST.MF
         299 Wed Oct 30 10:52:08 GMT 2024 META-INF/2024OVCS.SF
        5017 Wed Oct 30 10:52:08 GMT 2024 META-INF/2024OVCS.RSA
           0 Wed Jan 15 14:43:32 GMT 2020 META-INF/
sm       446 Wed Jan 15 14:21:40 GMT 2020 HelloWorld.class

  s = signature was verified 
  m = entry is listed in manifest
  k = at least one certificate was found in keystore

- Signed by "[REMOVED]"
    Digest algorithm: SHA256
    Signature algorithm: SHA256withRSA, 3072-bit key

jar verified.`

This is good.

Extracting the signed jar and running META-INF/2024OVCS.RSA through openssl:

[removed]@[removed] AzureKeyVault % openssl pkcs7 -in extracted/META-INF/2024OVCS.RSA -inform DER -print | grep subject: -A6
          subject: [REMOVED - DN OF SIGNING CERT]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 01 8a 02 82 01 81-00 b6 e2 bc 72 52   0...........rR
--
          subject: [REMOVED - DN OF INTERMEDIATE CA]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 02 0a 02 82 02 01-00 97 bc 2f d6 63   0........../.c
--
          subject: [REMOVED - DN OF ROOT CA]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 02 0a 02 82 02 01-00 e9 3d 47 f7 b1   0.........=G..

Notice the order: LEAF => INTERMEDIATE => ROOT. The signed CSR was imported in to a Key Vault as merge operation. The PEM file used contained the full chain in this order: ROOT => INTERMEDIATE => LEAF.

If I do the same process as above for a HSM backed key using the same openssl multi-tier CA we get different results. The output of verify is:

[removed]@[removed] AzureKeyVault % jarsigner -verify helloworld.signed.jar -verbose

         166 Wed Oct 30 11:00:32 GMT 2024 META-INF/MANIFEST.MF
         299 Wed Oct 30 11:00:32 GMT 2024 META-INF/SSMULTI.SF
        5014 Wed Oct 30 11:00:32 GMT 2024 META-INF/SSMULTI.RSA
           0 Wed Jan 15 14:43:32 GMT 2020 META-INF/
 m  ?    446 Wed Jan 15 14:21:40 GMT 2020 HelloWorld.class

  s = signature was verified 
  m = entry is listed in manifest
  k = at least one certificate was found in keystore
  ? = unsigned entry

- Signed by "[REMOVED - DN OF INTERMEDIATE CA]"
    Digest algorithm: SHA256
    Signature algorithm: SHA256withRSA, 4096-bit key

This is bad. It has attempted to sign with the intermediate CA and not the leaf.

Extracting the signed jar and running at META-INF/SSMULTI.RSA file through openssl:

[removed]@[removed] AzureKeyVault % openssl pkcs7 -in extracted/META-INF/SSMULTI.RSA -inform DER -print | grep subject: -A6
          subject: [REMOVED - DN OF SIGNING CERT]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 01 8a 02 82 01 81-00 99 b0 01 de 9d   0.............
--
          subject: [REMOVED - DN IF INTERMEDIATE CA]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 02 0a 02 82 02 01-00 97 bc 2f d6 63   0........../.c
--
          subject: [REMOVED - DN OF ROOT CA]
          key:           X509_PUBKEY: 
            algor: 
              algorithm: rsaEncryption (1.2.840.113549.1.1.1)
              parameter: NULL
            public_key:  (0 unused bits)
              0000 - 30 82 02 0a 02 82 02 01-00 e9 3d 47 f7 b1   0.........=G..

Its all in the same order...

The issue seems to be: the signing operation is picking the wrong certificate when it is a HSM backed key.

[moarychan]

@3millionminds did this work for you?

Hello,

Unfortunately we have been unable to test. We are unable to merge new certificates on top of the ones we already have in Azure Key Vault.

We have opened a MS ticket to see what options we have here.

@3millionminds is your problem solved?

Once a certificate operation is completed, you cannot merge another certificate (exported by DigiCert). If you need to merge again, please re-create the certificate in Key Vault, re-apply on DigiCert and then merge.

[moarychan]

Hi @dacron , thanks for your update!

If you use a key of type RSA-HSM and a DigiCert signed certificate it does not work. It doesn't work if I self sign a RSA-HSM backed cert either via the previously mentioned openssl-based CA. The verify of the DigiCert RSA-HSM cert shows:

I cannot generate a certificate with key type RSA-HSM on Azure Portal,

For a premium Key Vault, it supports to generate a key with key type RSA-HSM,

but how to generate a certificate using an existing key (RSA-HSM type)? Could you help provider more detail steps to create a cert chain with key type RSA-HSM on Azure Key Vault? This will be very helpful to verify this scenario.

[dacron]

We are using Premium Key Vault Vault:

We then send the resulting CSR over the DigiCert and merge the signed certificate back.

[moarychan]

Strange, I am using premium Key Vault, but I cannot see the Key Type with RSA-HSM when generating certificate.

Did you do any other settings?

[saragluna]

@moarychan I can help you with this

[moarychan]

Hi @dacron, thanks for your update!

I can use the private key type RSA-HSM and key size 3072 to generate a cert and signed with DiGiCert, it works well, could you follow below steps to retry?

1. Select key type RSA-HSM (when select No for exportable private key the RSA-HSM is avaiable) and key size 3072 to generate a cert in Azure portal, then select Create.
   

2. Download CSR file.
   

3. Sign on DiGiCert and download the single file containing all the certs.
   

4. Manuualy reorder the certs
   View the cert file
   

   Reorder, ROOT => INTERMEDIATE => LEAF
   

5. Merge the signed cert
   

6. Sign jar

   jarsigner \
   -keystore NONE \
   -storetype AzureKeyVault \
   -signedjar signerjar.jar unsigned.jar rsa-hsm-3072-reorder-certs \
   -verbose  \
   -storepass "" \
   -providerName AzureKeyVault \
   -providerClass com.azure.security.keyvault.jca.KeyVaultJcaProvider \
   -debug \
   -J--module-path="/xxx/com/azure/azure-security-keyvault-jca/2.10.0-beta.1/azure-security-keyvault-jca-2.10.0-beta.1.jar" \
   -J--add-modules="com.azure.security.keyvault.jca" \
   -J-Dazure.keyvault.uri=https://xxxkeyvaulthsm.vault.azure.net/ \
   -J-Dazure.keyvault.tenant-id=xxx \
   -J-Dazure.keyvault.client-id=xxx \
   -J-Dazure.keyvault.client-secret=xxx \
   -J-Djava.security.debug=jar

   

7. Verify jar

   jarsigner -verify -verbose -J-Djava.security.debug=jar -certs signerjar.jar

   

[dacron]

This works with the beta release of 2.10! Is it expected to work with 2.9.0 at all?

[rujche]

Hi, @dacron
By saying beta release of 2.10, do you mean beta version of 2.10 packaged by yourself?

I can't see 2.10 it in maven repository:
https://repo1.maven.org/maven2/com/azure/azure-security-keyvault-jca/

Now I plan to release 2.9.1 2.10.0 with this bug fixed. Current plan is to release 2.9.1 2.10.0 around the end of this month.

[dacron]

Yes - I built 2.10.0 beta 1 from source. Getting 2.9.1 out of the door would be much appreciated.

[dacron]

@rujche are you able to let me know which commit fixes the bug?

[rujche]

@dacron This one: #42522 #41303

[tanzim]

@dacron This one: #42522 #41303

Is that the correct PR? It doesn't make a lot of sense on surface at least, unless there's key vaults on MSFT side with certificates in name?

[rujche]

@dacron
Sorry, my mistake. It should be this one: #41303

[tanzim]

@dacron Sorry, my mistake. It should be this one: #41303

This implies 2.9 beta 2 should have the change? Nope it's not in the beta.

[rujche]

@dacron
Our plan updated. Here is the new plan:

1. Release 2.10.0 around the end of this month.
2. No patch release (2.9.1).

[moarychan]

JCA 2.10.0 is available on Maven Central, please try it! Thanks!