Compile tee-worker against rust-sgx-sdk v2.0

[Kailai-Wang]

Context

After a few weeks' intermittent work, finally, we may rejoice.

Important

This PR is ready to be reviewed and merged, but I suggest that we wait until state provisioning is tested on some host.

In theory, the change in this PR doesn't affect the client, but I label it as C1 to mark as a hint to any unexpected error (e.g. SEGV) that might arise.

Why do we need a new SDK

The main reasons are:

• blocking: newer polkadot/substrate dependencies need a newer rustc version, which isn't supported by the current sdk (v1.1.6). It blocks our further version update.
• non-blocking but nice-to-have: some crates (e.g. serde_json) have now no_std support, however, they need a newer rustc too. It's preferred to use no-std crate over ported sgx version of crates.
• sdk-v2.0 provides other benefits, please check their README

SDK

I'm using v2.0.0-preview branch of rust sgx sdk, this branch is not "officially" released but has been there for a while. It has been sporadically worked on, the unittests and samplecode are provided too - I tested most of them - looking good.

I do need to make some small adjustments in my own fork, please check TOML files for overriding:

• .cpp file compiles under ubuntu20, but not on ubutu22 due to a different (newer) gcc verison
• pointer misalignment fix

I'm considering submitting them as upstream PRs.
The associated rust-toolchain version is nightly-2023-11-17, but I believe it's possible to further upgrade it.

Other SDK-related files

I basically follow the example code in the sdk-v2.0 for the following files:

• Makefile (except I don't touch anything regarding BUILD_STD)
• Enclave configuration
• *.edl files and inc for enclave

Other SGX ported crates

I need to manually fork a bunch of sgx-ported crates to pair with the sdk-v2.0 (mainly sgx_tstd dep): https://github.com/Kailai-Wang?tab=repositories&q=sgx

They are under my personal workspace for now but can be migrated to litentry/ org after we obversing that they work well.

Among those crates, once_cell is the (only?) one that has no_std support but I still use the ported version, because it demands some critical-section implementation, which is unclear under sgx - I haven't found any either.

Note

The following are the problems / issues that I've met during the re-anchoring, some are resolved, some are work-arounded, some still need further investigation.

Intel SGX-SDK version

I have to use Intel sgx-sdk version 2.17, otherwise you'll get unmatched error when signing the enclave, see apache/incubator-teaclave-sgx-sdk#387

But in general, it should be possible to adjust it for newer sdk - we can have follow-up issues. We also need to test HW mode on a real host (preferrably ubuntu22.04) to see how it goes. Both CI and my local tests are conducted in SW mode.

De-/serialization of Rsa3072KeyPair

In sdk-v2.0, it implements sgx_serialize::{Deserialize, Serialize}. I was thinking about replacing it with no_std serde but eventually ditched it, mainly because:

• it seems unfavorable to depend on third party crates within the sdk
• sgx_serialize is widely used in sdk for other structs too, so I didn't mess around with it (in case others depend on the de-/serialization result of it)

As a result, I use sgx_serialize::json::{encode, decode} to deal with Rsa3072KeyPair de/ser. It should produce the same result as before and our ts-test should prove that.

SgxFile could be unstable

SgxFile is moved to sgx_tprotected_fs::SgxFile, there's a new HostFile struct. These seem to incur some problems with SgxFile creation/open/read:

• pointer missalignment in u_sgxfs_open_ocall - see fix above
• Previously generated sealed files can't be read directly by the new enclave - even for the same MRSIGNER and on the same host. I get

[ERROR enclave_runtime::error] Returning error Crypto(IO(Os { code: 2, kind: NotFound, message: "No such file or directory" })) as sgx unexpected.

even though the file is there. This must have something to do with the new HostFile impl. We can further look into this, but we should always be able to provision the state via RPC.

Use of USE_OPT_LIBS ?= 0

I have to use USE_OPT_LIBS ?= 0, otherwise sgx_crypto_sys and sgx_tlibc_sys complain prebuilt libs can't be found.
There's optimized_libs/external/libirc/efi2/libirc.a but not libm* => maybe we can further improve this.

Use of SGX_DEBUG=1

As you can see in Makefile, I use SGX_DEBUG=1 for the enclave compilation, and strip the symbols in the end to reduce the .so size. If compiled with SGX_DEBUG=0 - which basically implies release profile for cargo - the enclave will emit a SEGV during initialization, I've narrowed it down to this line:

report: Box::new(*Report::get_self()),

I didn't look into it as it could lie in somewhere deep (like compiler or fundamental crates/impl). Maybe it will be self-resolved as we upgrade them, but I can ask in upstream repo again

Use of cargo update

Most crates can be upgraded with a few exceptions:

• rustls-pki-types => has to be pinned at 0.2.1
• substrate-bip39 => has to be pinned at 0.4.4

They could be self-resolved when we upgrade the crate/tc version.

Tip

So given that 371 files are changed, how to review this PR?

Mainly these files:

• Makefile
• TOML files, especially tee-worker/Cargo.toml and tee-worker/enclave-runtime/Cargo.toml
• .rs files with non-trivial changes, by "trivial" I mean renaming like sgx_status_t => SgxStatus.

[linear]

P-430 Try out rust-sgx-sdk 2.0.0 branch

[Kailai-Wang]

It's not guaranteed that I wrote out all noticeable things - if I have something later I'll add it.

[kziemianek]

Rust code looks fine, nice job.

[Kailai-Wang]

2.21 SDK update is blocked by apache/incubator-teaclave-sgx-sdk#456 (mainly CLI tools)