Clarification on mr_modes

[traxys]

There is a bit of an edge case I'm not sure about. If I understand correctly modes are values that are set by the provider, and the application must support them in order for the provider to load, right ?

In other words the modes force the applications to do some stuff. If the application does not declare the mode then the getinfo call must fail with ENODATA I believe.

I see that some providers use the ofi_check_mr_mode function to perform this check, but it seems to be that the function does not check all modes (specifically FI_MR_ENDPOINT & FI_MR_RMA_EVENT). I think this results in a application succesfully loading without the modes if provider relies on the utils functions to perform those checks (with ofi_domain_check for example). Is this correct, and would this mean that a provider should manually check the MR modes ?

[shefty]

Your understanding of mr_mode is correct. The provider has a set of mr_mode bits that it requires of the app. The app indicates which mr_mode bits it can support. If the app does not set a specific mr_mode bit required by the provider, the provider will usually fail with ENODATA. (The provider doesn't have to fail, if it can work the bit being cleared. For example, by registering memory on demand, possibly using a MR cache, or by copying user data into pre-registered buffers.)

ofi_check_mr_mode() handles different libfabric versions, so the explicit checks that you see there are related to differences between libfabric 1.5 and earlier. This is the section of that function which is most applicable to updated checks:

			prov_mode = ofi_cap_mr_mode(user_info->caps, prov_mode);
			if (user_mode != FI_MR_UNSPEC &&
			    (user_mode & prov_mode) != prov_mode)
				goto out;

[traxys]

Okay thank you very much!