Configuration of libfabric+infiniband+ipoib #7885

theWayofthecode · 2022-07-15T15:22:16Z

theWayofthecode
Jul 15, 2022

Hi,

I followed the instruction of the vendor in order to setup a testbed for our libfabric development and I get a "connection refused" error. I'm pretty sure this is not a problem with libfabric but a misconfiguration of the system, so I would appreciate an advice of how to debug this.

fi_pingpong works in general but when I chose -p "verbs;ofi_rxm" it returns
fi_getinfo(): util/pingpong.c:1433, ret=-61 (No data available).

For the destination address I write the ip assigned to the infiniband interface by ipoib. So my fi_pingpong client invocation looks like this:

fi_pingpong -p "verbs;ofi_rxm" 172.31.0.253

and ifconfig returns:

ifconfig mlx4_ib0
mlx4_ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 65520
        inet 172.31.0.253  netmask 255.255.255.0  broadcast 172.31.0.255
        inet6 fe80::202:c903:b:c8c3  prefixlen 64  scopeid 0x20<link>
Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8).
        infiniband 80:00:02:08:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  txqueuelen 256  (InfiniBand)
        RX packets 41  bytes 5109 (4.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29  bytes 4397 (4.2 KiB)
        TX errors 0  dropped 18 overruns 0  carrier 0  collisions 0

This is the log of the server:


libfabric:core:core:fi_param_define_():223<info> registered var perf_cntr
libfabric:core:core:fi_param_get_():272<info> variable perf_cntr=<not set>
libfabric:core:core:fi_param_define_():223<info> registered var hook
libfabric:core:core:fi_param_get_():272<info> variable hook=<not set>
libfabric:core:core:fi_param_define_():223<info> registered var provider
libfabric:core:core:fi_param_define_():223<info> registered var fork_unsafe
libfabric:core:core:fi_param_define_():223<info> registered var universe_size
libfabric:core:core:fi_param_get_():272<info> variable provider=<not set>
libfabric:core:core:fi_param_define_():223<info> registered var provider_path
libfabric:core:core:fi_param_get_():272<info> variable provider_path=<not set>
libfabric:psm2:core:fi_psm2_ini():573<info> build options: HAVE_PSM2_SRC=0, HAVE_PSM2_AM_REGISTER_HANDLERS_2=1, HAVE_PSM2_MQ_FP_MSG=0, PSMX2_USE_REQ_CONTEXT=0
libfabric:psm2:core:fi_param_define_():223<info> registered var name_server
libfabric:psm2:core:fi_param_define_():223<info> registered var tagged_rma
libfabric:psm2:core:fi_param_define_():223<info> registered var uuid
libfabric:psm2:core:fi_param_define_():223<info> registered var delay
libfabric:psm2:core:fi_param_define_():223<info> registered var timeout
libfabric:psm2:core:fi_param_define_():223<info> registered var conn_timeout
libfabric:psm2:core:fi_param_define_():223<info> registered var prog_interval
libfabric:psm2:core:fi_param_define_():223<info> registered var prog_affinity
libfabric:psm2:core:fi_param_define_():223<info> registered var inject_size
libfabric:psm2:core:fi_param_define_():223<info> registered var lock_level
libfabric:psm2:core:fi_param_define_():223<info> registered var lazy_conn
libfabric:psm2:core:fi_param_define_():223<info> registered var disconnect
libfabric:psm2:core:fi_param_define_():223<info> registered var tag_layout
libfabric:psm2:core:fi_param_get_():272<info> variable name_server=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable tagged_rma=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable uuid=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable delay=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable timeout=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable conn_timeout=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable prog_interval=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable prog_affinity=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable inject_size=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable lock_level=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable lazy_conn=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable disconnect=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable tag_layout=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: psm2 (1.7)
libfabric:psm:core:fi_psm_ini():708<info>
libfabric:psm:core:fi_param_define_():223<info> registered var name_server
libfabric:psm:core:fi_param_define_():223<info> registered var am_msg
libfabric:psm:core:fi_param_define_():223<info> registered var tagged_rma
libfabric:psm:core:fi_param_define_():223<info> registered var uuid
libfabric:psm:core:fi_param_define_():223<info> registered var delay
libfabric:psm:core:fi_param_define_():223<info> registered var timeout
libfabric:psm:core:fi_param_define_():223<info> registered var prog_thread
libfabric:psm:core:fi_param_define_():223<info> registered var prog_interval
libfabric:psm:core:fi_param_define_():223<info> registered var prog_affinity
libfabric:psm:core:fi_param_get_():272<info> variable name_server=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable am_msg=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable tagged_rma=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable uuid=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable delay=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable timeout=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable prog_thread=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable prog_interval=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable prog_affinity=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: psm (1.7)
libfabric:core:core:ofi_register_provider():374<info> registering provider: usnic (1.0)
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():374<info> registering provider: shm (1.0)
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var buffer_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var comp_per_progress
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var sar_limit
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var use_srx
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var tx_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var rx_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var msg_tx_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var msg_rx_size
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable tx_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable rx_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable msg_tx_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable msg_rx_size=<not set>
libfabric:core:core:fi_param_get_():272<info> variable universe_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable buffer_size=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_rxm (1.0)
libfabric:verbs:core:fi_param_define_():223<info> registered var tx_size
libfabric:verbs:core:fi_param_get_():272<info> variable tx_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var rx_size
libfabric:verbs:core:fi_param_get_():272<info> variable rx_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var tx_iov_limit
libfabric:verbs:core:fi_param_get_():272<info> variable tx_iov_limit=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var rx_iov_limit
libfabric:verbs:core:fi_param_get_():272<info> variable rx_iov_limit=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var inline_size
libfabric:verbs:core:fi_param_get_():272<info> variable inline_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var min_rnr_timer
libfabric:verbs:core:fi_param_get_():272<info> variable min_rnr_timer=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var use_odp
libfabric:verbs:core:fi_param_get_():272<info> variable use_odp=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var prefer_xrc
libfabric:verbs:core:fi_param_get_():272<info> variable prefer_xrc=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var xrcd_filename
libfabric:verbs:core:fi_param_get_():272<info> variable xrcd_filename=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var cqread_bunch_size
libfabric:verbs:core:fi_param_get_():272<info> variable cqread_bunch_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var iface
libfabric:verbs:core:fi_param_get_():272<info> variable iface=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_cache_enable
libfabric:verbs:core:fi_param_get_():272<info> variable mr_cache_enable=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_max_cached_cnt
libfabric:verbs:core:fi_param_get_():272<info> variable mr_max_cached_cnt=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_max_cached_size
libfabric:verbs:core:fi_param_get_():272<info> variable mr_max_cached_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_cache_merge_regions
libfabric:verbs:core:fi_param_get_():272<info> variable mr_cache_merge_regions=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var dgram_use_name_server
libfabric:verbs:core:fi_param_get_():272<info> variable dgram_use_name_server=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var dgram_name_server_port
libfabric:verbs:core:fi_param_get_():272<info> variable dgram_name_server_port=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var gid_idx
libfabric:verbs:core:fi_param_get_():272<info> variable gid_idx=<not set>
libfabric:verbs:fabric:fi_ibv_get_device_attrs():555<info> The first found active port is 1
libfabric:verbs:fabric:fi_ibv_get_device_attrs():555<info> The first found active port is 1
libfabric:verbs:fabric:fi_ibv_get_device_attrs():555<info> The first found active port is 1
libfabric:core:core:ofi_register_provider():374<info> registering provider: verbs (1.0)
libfabric:ofi_mrail:core:fi_param_define_():223<info> registered var addr_strc
libfabric:ofi_mrail:core:fi_param_get_():272<info> variable addr_strc=<not set>
libfabric:ofi_mrail:core:mrail_parse_env_vars():63<warn> Unable to read OFI_MRAIL_ADDR_STRC env variable
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_mrail (1.0)
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var spin_count
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var retry
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var max_peers
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var max_unacked
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable spin_count=<not set>
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable retry=<not set>
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable max_peers=<not set>
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable max_unacked=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_rxd (1.0)
libfabric:core:core:ofi_register_provider():374<info> registering provider: UDP (1.1)
libfabric:sockets:core:fi_param_define_():223<info> registered var pe_waittime
libfabric:sockets:core:fi_param_define_():223<info> registered var conn_timeout
libfabric:sockets:core:fi_param_define_():223<info> registered var max_conn_retry
libfabric:sockets:core:fi_param_define_():223<info> registered var def_conn_map_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var def_av_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var def_cq_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var def_eq_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var pe_affinity
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_enable
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_time
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_intvl
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_probes
libfabric:sockets:core:fi_param_define_():223<info> registered var iface
libfabric:core:core:ofi_register_provider():374<info> registering provider: sockets (2.0)
libfabric:tcp:core:fi_param_define_():223<info> registered var iface
libfabric:core:core:ofi_register_provider():374<info> registering provider: tcp (0.1)
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_perf_hook (1.0)
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_noop_hook (1.0)
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable use_srx=<not set>
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:core:fi_ibv_check_hints():229<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_check_hints():230<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM
libfabric:verbs:core:fi_ibv_check_hints():230<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_REMOTE_READ
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:core:core:fi_fabric_():1080<info> Opened fabric: IB-0xfe80000000000000
libfabric:core:core:fi_fabric_():1080<info> Opened fabric: IB-0xfe80000000000000
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable use_srx=<not set>
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:core:fi_ibv_check_hints():229<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_check_hints():230<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM
libfabric:verbs:core:fi_ibv_check_hints():230<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:ofi_rxm:av:util_av_init():451<info> AV size 1
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable comp_per_progress=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable use_srx=<not set>
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:core:fi_ibv_check_hints():229<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_check_hints():230<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM
libfabric:verbs:core:fi_ibv_check_hints():230<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable sar_limit=<not set>
libfabric:ofi_rxm:core:rxm_ep_settings_init():2322<info> Settings:
                 MR local: MSG - 1, RxM - 1
                 Completions per progress: MSG - 1
                 Buffered min: 88
                 Min multi recv size: 16320
                 FI_EP_MSG provider inject size: 256
                 rxm inject size: 16320
                 Protocol limits: Eager: 16320, SAR: 131072
libfabric:verbs:core:fi_ibv_pep_listen():481<info> Listening on 172.31.0.253:9228

and this is the log of the client


[asi15@eu-muc-rnd088 Release]$ env FI_LOG_LEVEL=info fi_rma_bw -p "verbs;ofi_rxm" 172.31.0.253
libfabric:core:core:fi_param_define_():223<info> registered var perf_cntr
libfabric:core:core:fi_param_get_():272<info> variable perf_cntr=<not set>
libfabric:core:core:fi_param_define_():223<info> registered var hook
libfabric:core:core:fi_param_get_():272<info> variable hook=<not set>
libfabric:core:core:fi_param_define_():223<info> registered var provider
libfabric:core:core:fi_param_define_():223<info> registered var fork_unsafe
libfabric:core:core:fi_param_define_():223<info> registered var universe_size
libfabric:core:core:fi_param_get_():272<info> variable provider=<not set>
libfabric:core:core:fi_param_define_():223<info> registered var provider_path
libfabric:core:core:fi_param_get_():272<info> variable provider_path=<not set>
libfabric:psm2:core:fi_psm2_ini():573<info> build options: HAVE_PSM2_SRC=0, HAVE_PSM2_AM_REGISTER_HANDLERS_2=1, HAVE_PSM2_MQ_FP_MSG=0, PSMX2_USE_REQ_CONTEXT=0
libfabric:psm2:core:fi_param_define_():223<info> registered var name_server
libfabric:psm2:core:fi_param_define_():223<info> registered var tagged_rma
libfabric:psm2:core:fi_param_define_():223<info> registered var uuid
libfabric:psm2:core:fi_param_define_():223<info> registered var delay
libfabric:psm2:core:fi_param_define_():223<info> registered var timeout
libfabric:psm2:core:fi_param_define_():223<info> registered var conn_timeout
libfabric:psm2:core:fi_param_define_():223<info> registered var prog_interval
libfabric:psm2:core:fi_param_define_():223<info> registered var prog_affinity
libfabric:psm2:core:fi_param_define_():223<info> registered var inject_size
libfabric:psm2:core:fi_param_define_():223<info> registered var lock_level
libfabric:psm2:core:fi_param_define_():223<info> registered var lazy_conn
libfabric:psm2:core:fi_param_define_():223<info> registered var disconnect
libfabric:psm2:core:fi_param_define_():223<info> registered var tag_layout
libfabric:psm2:core:fi_param_get_():272<info> variable name_server=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable tagged_rma=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable uuid=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable delay=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable timeout=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable conn_timeout=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable prog_interval=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable prog_affinity=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable inject_size=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable lock_level=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable lazy_conn=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable disconnect=<not set>
libfabric:psm2:core:fi_param_get_():272<info> variable tag_layout=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: psm2 (1.7)
libfabric:psm:core:fi_psm_ini():708<info>
libfabric:psm:core:fi_param_define_():223<info> registered var name_server
libfabric:psm:core:fi_param_define_():223<info> registered var am_msg
libfabric:psm:core:fi_param_define_():223<info> registered var tagged_rma
libfabric:psm:core:fi_param_define_():223<info> registered var uuid
libfabric:psm:core:fi_param_define_():223<info> registered var delay
libfabric:psm:core:fi_param_define_():223<info> registered var timeout
libfabric:psm:core:fi_param_define_():223<info> registered var prog_thread
libfabric:psm:core:fi_param_define_():223<info> registered var prog_interval
libfabric:psm:core:fi_param_define_():223<info> registered var prog_affinity
libfabric:psm:core:fi_param_get_():272<info> variable name_server=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable am_msg=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable tagged_rma=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable uuid=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable delay=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable timeout=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable prog_thread=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable prog_interval=<not set>
libfabric:psm:core:fi_param_get_():272<info> variable prog_affinity=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: psm (1.7)
libfabric:core:core:ofi_register_provider():374<info> registering provider: usnic (1.0)
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():367<warn> no provider structure or name
libfabric:core:core:ofi_register_provider():374<info> registering provider: shm (1.0)
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var buffer_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var comp_per_progress
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var sar_limit
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var use_srx
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var tx_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var rx_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var msg_tx_size
libfabric:ofi_rxm:core:fi_param_define_():223<info> registered var msg_rx_size
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable tx_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable rx_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable msg_tx_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable msg_rx_size=<not set>
libfabric:core:core:fi_param_get_():272<info> variable universe_size=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable buffer_size=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_rxm (1.0)
libfabric:verbs:core:fi_param_define_():223<info> registered var tx_size
libfabric:verbs:core:fi_param_get_():272<info> variable tx_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var rx_size
libfabric:verbs:core:fi_param_get_():272<info> variable rx_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var tx_iov_limit
libfabric:verbs:core:fi_param_get_():272<info> variable tx_iov_limit=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var rx_iov_limit
libfabric:verbs:core:fi_param_get_():272<info> variable rx_iov_limit=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var inline_size
libfabric:verbs:core:fi_param_get_():272<info> variable inline_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var min_rnr_timer
libfabric:verbs:core:fi_param_get_():272<info> variable min_rnr_timer=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var use_odp
libfabric:verbs:core:fi_param_get_():272<info> variable use_odp=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var prefer_xrc
libfabric:verbs:core:fi_param_get_():272<info> variable prefer_xrc=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var xrcd_filename
libfabric:verbs:core:fi_param_get_():272<info> variable xrcd_filename=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var cqread_bunch_size
libfabric:verbs:core:fi_param_get_():272<info> variable cqread_bunch_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var iface
libfabric:verbs:core:fi_param_get_():272<info> variable iface=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_cache_enable
libfabric:verbs:core:fi_param_get_():272<info> variable mr_cache_enable=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_max_cached_cnt
libfabric:verbs:core:fi_param_get_():272<info> variable mr_max_cached_cnt=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_max_cached_size
libfabric:verbs:core:fi_param_get_():272<info> variable mr_max_cached_size=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var mr_cache_merge_regions
libfabric:verbs:core:fi_param_get_():272<info> variable mr_cache_merge_regions=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var dgram_use_name_server
libfabric:verbs:core:fi_param_get_():272<info> variable dgram_use_name_server=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var dgram_name_server_port
libfabric:verbs:core:fi_param_get_():272<info> variable dgram_name_server_port=<not set>
libfabric:verbs:core:fi_param_define_():223<info> registered var gid_idx
libfabric:verbs:core:fi_param_get_():272<info> variable gid_idx=<not set>
libfabric:verbs:fabric:fi_ibv_get_device_attrs():555<info> The first found active port is 1
libfabric:verbs:fabric:fi_ibv_get_device_attrs():555<info> The first found active port is 1
libfabric:verbs:fabric:fi_ibv_get_device_attrs():555<info> The first found active port is 1
libfabric:core:core:ofi_register_provider():374<info> registering provider: verbs (1.0)
libfabric:ofi_mrail:core:fi_param_define_():223<info> registered var addr_strc
libfabric:ofi_mrail:core:fi_param_get_():272<info> variable addr_strc=<not set>
libfabric:ofi_mrail:core:mrail_parse_env_vars():63<warn> Unable to read OFI_MRAIL_ADDR_STRC env variable
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_mrail (1.0)
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var spin_count
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var retry
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var max_peers
libfabric:ofi_rxd:core:fi_param_define_():223<info> registered var max_unacked
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable spin_count=<not set>
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable retry=<not set>
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable max_peers=<not set>
libfabric:ofi_rxd:core:fi_param_get_():272<info> variable max_unacked=<not set>
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_rxd (1.0)
libfabric:core:core:ofi_register_provider():374<info> registering provider: UDP (1.1)
libfabric:sockets:core:fi_param_define_():223<info> registered var pe_waittime
libfabric:sockets:core:fi_param_define_():223<info> registered var conn_timeout
libfabric:sockets:core:fi_param_define_():223<info> registered var max_conn_retry
libfabric:sockets:core:fi_param_define_():223<info> registered var def_conn_map_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var def_av_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var def_cq_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var def_eq_sz
libfabric:sockets:core:fi_param_define_():223<info> registered var pe_affinity
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_enable
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_time
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_intvl
libfabric:sockets:core:fi_param_define_():223<info> registered var keepalive_probes
libfabric:sockets:core:fi_param_define_():223<info> registered var iface
libfabric:core:core:ofi_register_provider():374<info> registering provider: sockets (2.0)
libfabric:tcp:core:fi_param_define_():223<info> registered var iface
libfabric:core:core:ofi_register_provider():374<info> registering provider: tcp (0.1)
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_perf_hook (1.0)
libfabric:core:core:ofi_register_provider():374<info> registering provider: ofi_noop_hook (1.0)
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable use_srx=<not set>
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:core:fi_ibv_check_hints():229<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_check_hints():230<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM
libfabric:verbs:core:fi_ibv_check_hints():230<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_REMOTE_READ
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: No such device(19)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:verbs:fabric:fi_ibv_get_rai_id():187<info> rdma_bind_addr: Invalid argument(22)
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:core:core:fi_fabric_():1080<info> Opened fabric: IB-0xfe80000000000000
libfabric:core:core:fi_fabric_():1080<info> Opened fabric: IB-0xfe80000000000000
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable use_srx=<not set>
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:core:fi_ibv_check_hints():229<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_check_hints():230<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM
libfabric:verbs:core:fi_ibv_check_hints():230<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:ofi_rxm:av:util_av_init():451<info> AV size 1
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable comp_per_progress=<not set>
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable use_srx=<not set>
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_rxd
libfabric:verbs:fabric:fi_ibv_set_default_attr():1111<info> Ignoring provider default value for tx rma_iov_limit as it is greater than the value supported by domain: mlx4_0
libfabric:verbs:fabric:fi_ibv_get_matching_info():1213<info> hints->ep_attr->rx_ctx_cnt != FI_SHARED_CONTEXT. Skipping XRC FI_EP_MSG endpoints
libfabric:verbs:core:fi_ibv_check_hints():229<info> Unsupported capabilities
libfabric:verbs:core:fi_ibv_check_hints():230<info> Supported: FI_MSG, FI_RECV, FI_SEND, FI_LOCAL_COMM, FI_REMOTE_COMM
libfabric:verbs:core:fi_ibv_check_hints():230<info> Requested: FI_MSG, FI_RMA, FI_READ, FI_WRITE, FI_RECV, FI_SEND, FI_REMOTE_READ, FI_REMOTE_WRITE
libfabric:core:core:ofi_layering_ok():790<info> Need core provider, skipping ofi_mrail
libfabric:ofi_rxm:core:fi_param_get_():272<info> variable sar_limit=<not set>
libfabric:ofi_rxm:core:rxm_ep_settings_init():2322<info> Settings:
                 MR local: MSG - 1, RxM - 1
                 Completions per progress: MSG - 1
                 Buffered min: 88
                 Min multi recv size: 16320
                 FI_EP_MSG provider inject size: 256
                 rxm inject size: 16320
                 Protocol limits: Eager: 16320, SAR: 131072
libfabric:verbs:core:fi_ibv_pep_listen():481<info> Listening on 172.31.0.253:55392
libfabric:verbs:core:ofi_check_domain_attr():525<info> Unknown domain name
libfabric:verbs:core:ofi_check_domain_attr():526<info> Supported: mlx4_0-xrc
libfabric:verbs:core:ofi_check_domain_attr():526<info> Requested: mlx4_0
libfabric:verbs:core:ofi_check_ep_type():629<info> Unsupported endpoint type
libfabric:verbs:core:ofi_check_ep_type():630<info> Supported: FI_EP_DGRAM
libfabric:verbs:core:ofi_check_ep_type():630<info> Requested: FI_EP_MSG
libfabric:verbs:fabric:fi_ibv_create_ep():224<info> rdma_create_ep: Connection timed out(110)
libfabric:ofi_rxm:ep_ctrl:rxm_msg_ep_open():819<warn> Unable to create msg_ep
transmit(): common/shared.c:1649, ret=-110 (Connection timed out)
libfabric:psm2:core:psmx2_fini():538<info>
libfabric:psm:core:psmx_fini():677<info>
libfabric:usnic:fabric:usdf_fini():1245<trace>

Do you have a hint about where to look for problems?

Thanks,
Aram

ghost · 2022-07-15T15:42:44Z

ghost
Jul 15, 2022

please share ifconfig info for both sides.

0 replies

j-xiong · 2022-07-15T15:47:39Z

j-xiong
Jul 15, 2022
Maintainer

Also please check the output of ibv_devinfo and fi_info -p verbs.

0 replies

theWayofthecode · 2022-08-02T09:14:14Z

theWayofthecode
Aug 2, 2022
Author

Hi,

sorry for the delayed response, I was on vacations.

Right now the ibv_devinfo shows that the PORT is not ACTIVE (just INIT(2)). This was not the case before when I made the post, so right now there exist a more fundamental problem. I'll fix it and get our system in a proper state and if the original problem persists, I'll get back to you ASAP.

Thanks again,
Aram

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuration of libfabric+infiniband+ipoib #7885

{{title}}

Replies: 3 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Configuration of libfabric+infiniband+ipoib #7885

theWayofthecode Jul 15, 2022

Replies: 3 comments

ghost Jul 15, 2022

j-xiong Jul 15, 2022 Maintainer

theWayofthecode Aug 2, 2022 Author

theWayofthecode
Jul 15, 2022

ghost
Jul 15, 2022

j-xiong
Jul 15, 2022
Maintainer

theWayofthecode
Aug 2, 2022
Author