MCUboot refactoring directions and ideas

[davidvincze]

Some opening thoughts (non-exhaustive) from previous discussions.

Configuration topic:

I do not even think people try to figure out whether Mbed TLS or TinyCrypt will be used because it is hard to figure out which encryption/signature will compile with which library. ...... configuration requires magic skills: if you have preferred lib, or it is provided by system anyway, it is hard impossible to figure out combination of options to build. And it is not possible to have mix, maybe somebody would like for some reason, where you have ECB encryption and RSA signature because then you have lib collision.
............................................................
I already thought about a feature matrix in the documentation, but an automatic config check would be much better.
MCUboot configuration happens (officially) through the mcuboot_config.h header, but I've seen KConfig and CMake... etc. for being used as a configuration backend, making the header file secondary. The combination of a proper documentation, updated sample_config_file and a basic, centralized/decentralized config check (already present in some of the modules) could be a good start.
............................................................
The problem with mcuboot_config.h is that there is separate one for every system, while there should be the main one that defines all available MCUBOOT_ type identifiers, then includes system/platform specific overrides, which redefine some of them, which is then followed by some checks. We probably need autoconf type C file to be build to actually figure out some stuff but that is impossible wish at this time.

Code structure topic:

There is ifdef party in headers like sha256.h, while we should have some plug-in architecture for crypto with MCUboot headers defining the API, we have a mess. Now if you would like to support some other lib or maybe hardware accelerated Mbed TLS there are more ifdefs going to happen in sha256.h and other headers.
Or even if you want to support some two incompatible releases of Mbed TLS, to give people grace period to switch, it is a lot of additional ifdefs.
............................................................
The loader.c seems like a lot of work the divide and give the core logic a much more modular structure. The crypto modules are a much easier target. I think we should identify code parts that could be rearranged thus making the modularization of the remaining parts easier.

A few newer PRs related to code structure refactoring:

• encryption: bootutil: Refactor encrypted.c to have the same logic as image_validate #1658
• RSA and SHA module: Add support for a PSA Crypto APIs backend for RSA and SHA #1660
• ECDSA module: Add a generic signature validation module for ECDSA #1666