Interoperability with libtomcrypt / OpenSSL

[MauriceKayser]

I am trying to migrate a libtomcrypt based C program, which performs the following operations:

1. Decrypt an AES-128 key with an RSA public key
2. Decrypt data with the decrypted AES-128 key
3. Verify the whole operation with RSA

It uses the following buffers:

• der_pub_key[u8; ...]
• enc_data[u8; ...]
• enc_aes_key[u8; 128]
• enc_rsa_sig[u8; 128]
• label[u8; 4]
• nonce[u8; 16]
• tag[u8; 16]

It executes the following methods (+ their logical equivalents in RustCrypto):

1. pub_key = rsa_import_pkcs1(der_pub_key) → rsa::RsaPublicKey::from_pkcs1_der
2. aes_key = rsa_exptmod(enc_aes_key, pub_key) → rsa::RsaPublicKey::raw_encryption_primitive
3. aes_key = pkcs_1_oaep_decode(aes_key, label, SHA256) → rsa::oaep::decrypt_inner w/o priv_key.raw_decryption_primitive
4. data = eax_decrypt_verify_memory(aes_key, nonce, label, enc_data, tag, AES128) → eax::Eax::<aes::Aes128>::decrypt
5. rsa_sig = rsa_exptmod(enc_rsa_sig, pub_key) → rsa::RsaPublicKey::raw_encryption_primitive
6. verified = pkcs_1_pss_decode(tag, rsa_sig, SHA256) → rsa::pss::emsa_pss_verify

I replicated the decrypted data & signature verification by ripping out some of the RustCrypto/RSA crate code, because I could not figure out how to use it otherwise:

Issue 1: RustCrypto does not allow public RSA keys to decrypt data

Step 2 & 3 should be combinable into: rsa::RsaPublicKey::decrypt(enc_aes_key, label, OAEP<SHA256>) - step 2 is implemented in rsa::RsaPublicKey, but step 3 is only implemented in rsa::RsaPrivateKey. rsa::RsaPublicKey does not expose a decrypt method, which is possible with libtomcrypt, and OpenSSL seems to support that as well.
I copied and combined the code of rsa::RsaPublicKey::raw_encryption_primitive and rsa::oaep::decrypt_inner (without the priv_key.raw_decryption_primitive part), to mimic the C program behaviour.

Issue 2: RustCrypto verification steps differ from libtomcrypt

Step 5 & 6 can be combined into: rsa::RsaPublicKey::verify(tag, rsa_sig, PSS<SHA256>). But it fails at the first validation step in rsa::pss::emsa_pss_verify:

// 1. If the length of M is greater than the input limitation for the
//    hash function (2^61 - 1 octets for SHA-1), output "inconsistent"
//    and stop.
//
// 2. Let mHash = Hash(M), an octet string of length hLen
let h_len = hash.output_size();
if m_hash.len() != h_len {
    return Err(Error::Verification);
}

In my case it equates to 16 != 32, as m_hash is the tag[u8; 16] buffer of the C program, which is used for both AES-128 and RSA with SHA2-256 (hash is SHA2-256). Changing the condition to if m_hash.len() > h_len, which could be a valid interpretation of the "1. If the length of M is greater than [..]" RFC-8017 description, the check and all other subsequent checks pass and attest the validity of the data, like the C program does. Though that interpretation contradicts "2. [..] string of length hLen" as far as I understand.
I think OpenSSL does not provide a possibility to set an explicit length (like 16) either, so I guess it assumes 32 as well. The OpenSSL Python version explicitly mentions, that they have no way of checking the condition. libtomcrypt did not enforce this for the authors of the C program the way RustCrypto does for me, so I need to copy and modify the rsa::RsaPublicKey::verify code.
I can not judge whether this is an issue with libtomcrypt checking too little, or RustCrypto checking too much.

Both issues would also affect the Golang code RustCrypto was inspired by.

[tarcieri]

It sounds like you're using a number of nonstandard combinations of primitives.

From what I can tell, you're attempting to roll your own version of signcryption, which allows decryption by the public key and also authenticates the message ala a typical digital signature.

There's a semi-standard RSA construction for this: Probabilistic Signature Scheme with Recovery (PSS-R) as described in Section 5 of: https://web.cs.ucdavis.edu/~rogaway/papers/exact.pdf

This approach allows for "folding" part of the message into the signature in such a way that it is "recoverable" by the verifier.

[MauriceKayser]

That does seem to be what is going on, thanks for the hint! Do you think the 2 mentioned issues will be addressed in this crate, or should I switch to OpenSSL / continue using the copied code pieces?

[tarcieri]

We could potentially add PSS-R as a high-level construction.

The low-level primitives you're describing are easily misused and I personally don't think it would be good to add them.

[dignifiedquire]

The low-level primitives you're describing are easily misused and I personally don't think it would be good to add them.

While I agree that they are easily misused, I have been often enough in the situation where I needed to support specific legacy constructions, that I wouldn't mind exposing this with a big warning sign under the hazmat flag.

We could potentially add PSS-R as a high-level construction.

👍

[tarcieri]

Closing in favor of #231 which discusses adding PSS-R support