Value of Public Exponent is longer than 5 bytes, Signature.isVerify will always return false.

[nonark]

Using Conscrypt Library Version

org.conscrypt:conscrypt-android:2.5.2

Descript

• If the value of Public Exponent is longer than 5 bytes, Signature.isVerify will always return false.
• When testing with the same test value in another library, it works normally.
• An exception occurs in the Cipher.doFinal step during the Signature test with "RSA/None/PKCS1Padding" and "SHA-256". (Sample Code 2)

Sample Data

String modulus = "BE4BCFD33758FBB0092B204AB9B8B7ACD1F0CDDFE18AEB8EB472D8F82D669C749622257DD22A4D7481B20DAF6A09951C2404D43279BF29C83BB94E8D4FAAC15811E97D82121863CEA53F1E08AE1EDA6F7695B889EF536A9FAE21CCF4146F2397D834CAB245CB6402B9FAE9229AD193CF42CA586C0F87D42D09AFB2A2891D8910517CEF7EA92C35D1D78D051ED6FA11111E120E8E3789FEC533210AD8A290A1637A01D3EEA53169624487D678B1E9A5BCA59FB950D8534BCE9F38BB226C40BEC242A607602006D5B73B44901E104CB38E8972AAB28BAFC8640C33BD956A31BA995AAA978C849459092CB9825A114082AAEBA29AB270EA29D7F57990FA4C36AA99";
String exponent = "087CBA709F";
String message = "A000DB18631B5A3180667986ED5249B99EADA3863424EDECBF462E6BB421830B9B715DD7C38396FB08A7D9CB362A663EACD70626501CDCFED79E9CFA1663D95932AFB05491F591A5D3F4A21022C48A5B4145CF8D7BB5BB4CC6EECC54286AEFABB7FC90C085A8837B6582DB0C69A2ECA3F2C465060E9155C27036FC1470F627AF";
String signature = "6C6F4E238A9D600FB80101ABFF292CDC49400FBEFA5AD032BE00AFEF0C682A8A0D15B2CD14E917F3019A84476E715D2E0AFCDC953BCE00A543AD9A289C67A335EF92BBB6E0544A32D4C57CE1273C3A174BCA9285DCF3F7C2EFBA4060463F51B1E318945A5C74440343AB46A682423D6158A64C0D4B9B806198C775C892B124BCB996D57B28F15320297D173C2FEA59AEAE9A291CFF5532E56FEEA01140DB0D9C39533F1560408F974FAA79154D5C3ED73062727635DF19CCA6DA4B9F52DEFDA5526A5D357E1A1793C27382C0C749567BF6442F626D2365D5DD6F2E9A696DBF3D60F468AD63BF8F4B96915D0DA69DDCB5907D3089B9108D98770B2356BD974B46";

Sample Test Verify Result

OS	API (Provider)	Verify
Apple	RSASignatureMessagePKCS1v15SHA256	True
Android	StripyCastle(SCFIPS) Provider	True
Android	Conscrypt Provider	False

Sample Code 1

RSAPublicKeySpec rsaPublicKeySpec = new RSAPublicKeySpec(
    new BigInteger("BE4BCFD33758FBB0092B204AB9B8B7ACD1F0CDDFE18AEB8EB472D8F82D669C749622257DD22A4D7481B20DAF6A09951C2404D43279BF29C83BB94E8D4FAAC15811E97D82121863CEA53F1E08AE1EDA6F7695B889EF536A9FAE21CCF4146F2397D834CAB245CB6402B9FAE9229AD193CF42CA586C0F87D42D09AFB2A2891D8910517CEF7EA92C35D1D78D051ED6FA11111E120E8E3789FEC533210AD8A290A1637A01D3EEA53169624487D678B1E9A5BCA59FB950D8534BCE9F38BB226C40BEC242A607602006D5B73B44901E104CB38E8972AAB28BAFC8640C33BD956A31BA995AAA978C849459092CB9825A114082AAEBA29AB270EA29D7F57990FA4C36AA99", 16),
    new BigInteger("087CBA709F", 16)
);

KeyFactory keyFactory = KeyFactory.getInstance("RSA", "Conscrypt");
PublicKey rsaPublicKey = keyFactory.generatePublic(rsaPublicKeySpec);

Signature signature = Signature.getInstance("SHA256WithRSA", "Conscrypt");
signature.initVerify(rsaPublicKey);
signature.update(ByteUtil.hexStringToByteArray("A000DB18631B5A3180667986ED5249B99EADA3863424EDECBF462E6BB421830B9B715DD7C38396FB08A7D9CB362A663EACD70626501CDCFED79E9CFA1663D95932AFB05491F591A5D3F4A21022C48A5B4145CF8D7BB5BB4CC6EECC54286AEFABB7FC90C085A8837B6582DB0C69A2ECA3F2C465060E9155C27036FC1470F627AF"));

// isVerify Expected 'true', Result ' false'
boolean isVerify = signature.verify(ByteUtil.hexStringToByteArray("6C6F4E238A9D600FB80101ABFF292CDC49400FBEFA5AD032BE00AFEF0C682A8A0D15B2CD14E917F3019A84476E715D2E0AFCDC953BCE00A543AD9A289C67A335EF92BBB6E0544A32D4C57CE1273C3A174BCA9285DCF3F7C2EFBA4060463F51B1E318945A5C74440343AB46A682423D6158A64C0D4B9B806198C775C892B124BCB996D57B28F15320297D173C2FEA59AEAE9A291CFF5532E56FEEA01140DB0D9C39533F1560408F974FAA79154D5C3ED73062727635DF19CCA6DA4B9F52DEFDA5526A5D357E1A1793C27382C0C749567BF6442F626D2365D5DD6F2E9A696DBF3D60F468AD63BF8F4B96915D0DA69DDCB5907D3089B9108D98770B2356BD974B46"));

Sample Code 2

RSAPublicKeySpec rsaPublicKeySpec = new RSAPublicKeySpec(
    new BigInteger("BE4BCFD33758FBB0092B204AB9B8B7ACD1F0CDDFE18AEB8EB472D8F82D669C749622257DD22A4D7481B20DAF6A09951C2404D43279BF29C83BB94E8D4FAAC15811E97D82121863CEA53F1E08AE1EDA6F7695B889EF536A9FAE21CCF4146F2397D834CAB245CB6402B9FAE9229AD193CF42CA586C0F87D42D09AFB2A2891D8910517CEF7EA92C35D1D78D051ED6FA11111E120E8E3789FEC533210AD8A290A1637A01D3EEA53169624487D678B1E9A5BCA59FB950D8534BCE9F38BB226C40BEC242A607602006D5B73B44901E104CB38E8972AAB28BAFC8640C33BD956A31BA995AAA978C849459092CB9825A114082AAEBA29AB270EA29D7F57990FA4C36AA99", 16),
    new BigInteger("087CBA709F", 16)
);

KeyFactory keyFactory = KeyFactory.getInstance("RSA", "Conscrypt");
PublicKey publicKey = keyFactory.generatePublic(rsaPublicKeySpec);

Cipher cipher = Cipher.getInstance("RSA/None/PKCS1Padding", "Conscrypt");
cipher.init(Cipher.DECRYPT_MODE, publicKey);

// "BAD_E_VALUE" Exception Occurs. 
byte[] decSig = cipher.doFinal(ByteUtil.hexStringToByteArray("6C6F4E238A9D600FB80101ABFF292CDC49400FBEFA5AD032BE00AFEF0C682A8A0D15B2CD14E917F3019A84476E715D2E0AFCDC953BCE00A543AD9A289C67A335EF92BBB6E0544A32D4C57CE1273C3A174BCA9285DCF3F7C2EFBA4060463F51B1E318945A5C74440343AB46A682423D6158A64C0D4B9B806198C775C892B124BCB996D57B28F15320297D173C2FEA59AEAE9A291CFF5532E56FEEA01140DB0D9C39533F1560408F974FAA79154D5C3ED73062727635DF19CCA6DA4B9F52DEFDA5526A5D357E1A1793C27382C0C749567BF6442F626D2365D5DD6F2E9A696DBF3D60F468AD63BF8F4B96915D0DA69DDCB5907D3089B9108D98770B2356BD974B46"));

ASN1InputStream aIn = new ASN1InputStream(decSig);
ASN1Sequence seq = (ASN1Sequence) aIn.readObject();

ASN1OctetString sigHash = (ASN1OctetString) seq.getObjectAt(1);

MessageDigest hash = MessageDigest.getInstance("SHA-256");
hash.update(ByteUtil.hexStringToByteArray("BE4BCFD33758FBB0092B204AB9B8B7ACD1F0CDDFE18AEB8EB472D8F82D669C749622257DD22A4D7481B20DAF6A09951C2404D43279BF29C83BB94E8D4FAAC15811E97D82121863CEA53F1E08AE1EDA6F7695B889EF536A9FAE21CCF4146F2397D834CAB245CB6402B9FAE9229AD193CF42CA586C0F87D42D09AFB2A2891D8910517CEF7EA92C35D1D78D051ED6FA11111E120E8E3789FEC533210AD8A290A1637A01D3EEA53169624487D678B1E9A5BCA59FB950D8534BCE9F38BB226C40BEC242A607602006D5B73B44901E104CB38E8972AAB28BAFC8640C33BD956A31BA995AAA978C849459092CB9825A114082AAEBA29AB270EA29D7F57990FA4C36AA99"));

boolean isVerify = ByteUtil.byteArrayToHexString(hash.digest()).equals(ByteUtil.byteArrayToHexString(sigHash.getOctets()));

Sample Code 2 Result

javax.crypto.BadPaddingException: error:04000065:RSA routines:OPENSSL_internal:BAD_E_VALUE
    at org.conscrypt.NativeCrypto.RSA_public_decrypt(Native Method)
    at org.conscrypt.OpenSSLCipherRSA$DirectRSA.doCryptoOperation(OpenSSLCipherRSA.java:400)
    at org.conscrypt.OpenSSLCipherRSA.engineDoFinal(OpenSSLCipherRSA.java:312)
    at javax.crypto.Cipher.doFinal(Cipher.java:2055)

[prbprbprb]

On the one hand, this is expected behaviour. BoringSSL rejects any RSA public key where e > 33 bits long. See the references in the relevant check for a good background on why. In general, large values of e add computational complexity without adding security, and I believe BoringSSL would have chosen a (slightly) lower limit[1] except 2^32+1 is in use for DNSSEC by some TLDs.

On the other hand, the results you are getting here are buggy. In the first example Signature.initVerify() should have thrown InvalidKeyException and in the second Cipher.init() should have thrown it, so thanks for catching that!

[1] E.g. The Windows struct _RSAPUBKEY only allows 32 bits for e, so that provides a good upper bound for anything that needs to interoperate with Windows.

[prbprbprb]

Actually, thinking about it, the validity checks should probably go on the code paths which create public and private RSA keys either from KeySpec value constructors as above or PKCS#8 / X.509 encodings.

[davidben]

Yeah the validity checks being delayed is a consequence of bad OpenSSL APIs. If you create them from the parser, BoringSSL will check such things eagerly. But if you create from individual components, it's a mess. In a lot of cases, you have to call RSA_check_key. Except that's particularly tricky for Conscrypt because by default, RSA_check_key won't tolerate keys with n and d, but not e.

The new APIs from #1133 resolve this mess. Actually, one of the side effects is that PR + new BoringSSLwill check validity at key creation rather than deferring to use. (Sorry I should have called that out but forgot to.) So that part is all set, for newer BoringSSL. For older BoringSSL, you almost could turn on the RSA_check_key codepath, but you'd have to deal with the n , d thing, so might be easier to just get everything to new BoringSSL. :-/

[prbprbprb]

Actually, one of the side effects is that PR + new BoringSSL will check validity at key creation rather than deferring to use

Fab! We probably need some tests on the Conscrypt side (if only to validate the exception thrown), but I can take care of that.