TAO_UIPMC_Protocol_Factory's -ORBListenerInterfaces does not work with IPv6 addresses and has other issues

[nickwilliams-zaxiom]

Note: Submission of this bug will be shortly followed by submission of a pull request to fix this bug and add a new test for it.

Version

ACE 7.0.9
TAO 3.0.9

Host machine and operating system

• macOS 12, 13; Clang 13, 14, 15
• Ubuntu 20.04; GCC 9
• Ubuntu 22.04; GCC 11
• CentOS 7; GCC 11
• RHEL 7; GCC 11
• RHEL 8; GCC 8
• RHEL 8; GCC 11
• RHEL 9; GCC 11
• Windows 10; VS 2022 via Cygwin
• Windows 11; VS 2022 via Cygwin
• Windows Server 2022; VS 2022 via Cygwin

Target machine and operating system (if different from host)

n/a

Compiler name and version (including patch level)

See above; patch versions do not affect the outcome, because it's the same across every platform and compiler.

The $ACE_ROOT/ace/config.h file

Simplest possible one to replicate this problem:

#define ACE_HAS_IPV6 1
#define ACE_USES_IPV4_IPV6_MIGRATION 1

Our full file:

#define TAO_DEFAULT_MIOP_SEND_THROTTLING false

#define ACE_HAS_IPV6
#define ACE_USES_IPV4_IPV6_MIGRATION

#ifndef ACE_HAS_VERSIONED_NAMESPACE
#define ACE_HAS_VERSIONED_NAMESPACE 1
#endif /* ACE_HAS_VERSIONED_NAMESPACE */

#if __cplusplus >= 201703L
#define ACE_LACKS_AUTO_PTR // with C++17, auto_ptr is gone
#endif

#ifdef ACE_HAS_SCTP
#error ACE_HAS_SCTP is defined, but it should not be
#endif // To disable SCTP support, this cannot be defined to anything, even 0 (i.e., zero)

#define ACE_DOESNT_INSTANTIATE_NONSTATIC_OBJECT_MANAGER 1
#define ACE_DONT_INIT_WINSOCK 1
#define ACE_INITIALIZE_MEMORY_BEFORE_USE 1
#define ACE_HAS_EXCEPTIONS 1
#define ACE_HAS_MONITOR_FRAMEWORK 0
#define ACE_HAS_NONSTATIC_OBJECT_MANAGER 1
#define ACE_HAS_REACTOR_NOTIFICATION_QUEUE 1
#define ACE_HAS_STRICT 1
#define TAO_HAS_CORBALOC_PARSER 1
#define TAO_HAS_CORBANAME_PARSER 1
#define TAO_HAS_DDL_PARSER 0
#define TAO_HAS_DIOP 1
#define TAO_HAS_HTTP_PARSER 0
#define TAO_HAS_MCAST_PARSER 1
#define TAO_HAS_SCIOP 0
#define TAO_HAS_SHMIOP 0
#define TAO_HAS_UIOP 0
#define TAO_HAS_ZIOP 0
#define TAO_LACKS_EVENT_CHANNEL_ANY 1
#define TAO_LACKS_EVENT_CHANNEL_TIMESTAMPS 1
#define TAO_SUPPRESS_NEW_LINE_IN_EXCEPTION_LOGGING 1
#define TAO_USE_DOTTED_DECIMAL_ADDRESSES 1
#define ACE_DISABLE_MKTEMP 1 // the use of `mktemp' is dangerous, better use `mkstemp'
#define ACE_DISABLE_TEMPNAM 1 // 'tempnam' is deprecated: This function is provided for compatibility reasons only. Due to security
                              // concerns inherent in the design of tempnam(3), it is highly recommended that you use mkstemp(3) instead.
//#define TAO_ZERO_TAO_OUTPUTCDR_ALLOCATED_BUFFERS 1
//#define POA_NO_TIMESTAMP 1
//#define TAO_HAS_FILE_PARSER 1

// The service configurator tries to dynamically load things unless we define this
#define TAO_AS_STATIC_LIBS 1
#define ACE_AS_STATIC_LIBS 1

// Defining STATIC_LIBS (above) turns off the DLL export stuff.  Turn it back on.
#define ACEXML_HAS_DLL 1
#define ACE_FLREACTOR_HAS_DLL 1
#define ACE_FOXREACTOR_HAS_DLL 1
#define ACE_HAS_DLL 1
#define ACE_QTREACTOR_HAS_DLL 1
#define ACE_QoS_HAS_DLL 1
#define ACE_SSL_HAS_DLL 1
#define ACE_SVC_HAS_DLL 1
#define ACE_TKREACTOR_HAS_DLL 1
#define ACE_XTREACTOR_HAS_DLL 1
#define ACTIVATOR_HAS_DLL 1
#define ACTIVATOR_IDL_HAS_DLL 1
#define DYNSERVER_HAS_DLL 1
#define HTIOP_HAS_DLL 1
#define LOCATOR_HAS_DLL 1
#define LOCATOR_IDL_HAS_DLL 1
#define TAO_ANYTYPECODE_HAS_DLL 1
#define TAO_AV_HAS_DLL 1
#define TAO_BIDIRGIOP_HAS_DLL 1
#define TAO_BZIP2COMPRESSOR_HAS_DLL 1
#define TAO_CODECFACTORY_HAS_DLL 1
#define TAO_CODESET_HAS_DLL 1
#define TAO_COMPRESSION_HAS_DLL 1
#define TAO_CONCURRENCY_HAS_DLL 1
#define TAO_CONCURRENCY_SERV_HAS_DLL 1
#define TAO_CONCURRENCY_SKEL_HAS_DLL 1
#define TAO_CSD_FW_HAS_DLL 1
#define TAO_CSD_TP_HAS_DLL 1
#define TAO_DIFFSERVPOLICY_HAS_DLL 1
#define TAO_DYNAMICANY_HAS_DLL 1
#define TAO_DYNAMICINTERFACE_HAS_DLL 1
#define TAO_ENDPOINTPOLICY_HAS_DLL 1
#define TAO_ESF_HAS_DLL 1
#define TAO_EVENTLOG_HAS_DLL 1
#define TAO_EVENTLOG_SERV_HAS_DLL 1
#define TAO_EVENTLOG_SKEL_HAS_DLL 1
#define TAO_EVENT_HAS_DLL 1
#define TAO_EVENT_SERV_HAS_DLL 1
#define TAO_EVENT_SKEL_HAS_DLL 1
#define TAO_FLRESOURCE_HAS_DLL 1
#define TAO_FOXRESOURCE_HAS_DLL 1
#define TAO_FTRTEC_HAS_DLL 1
#define TAO_FTRTEVENT_HAS_DLL 1
#define TAO_FTRT_HAS_DLL 1
#define TAO_FT_CLIENTORB_HAS_DLL 1
#define TAO_FT_HAS_DLL 1
#define TAO_FT_ORB_UTILS_HAS_DLL 1
#define TAO_FT_SERVERORB_HAS_DLL 1
#define TAO_HAS_DLL 1
#define TAO_IDL_BE_HAS_DLL 1
#define TAO_IDL_FE_HAS_DLL 1
#define TAO_IFRSERVICE_HAS_DLL 1
#define TAO_IFR_BE_HAS_DLL 1
#define TAO_IFR_CLIENT_HAS_DLL 1
#define TAO_IMR_CLIENT_HAS_DLL 1
#define TAO_IORINTERCEPTOR_HAS_DLL 1
#define TAO_IORManip_HAS_DLL 1
#define TAO_IORTable_HAS_DLL 1
#define TAO_LIFECYCLE_HAS_DLL 1
#define TAO_LIFECYCLE_SKEL_HAS_DLL 1
#define TAO_LOADBALANCING_HAS_DLL 1
#define TAO_LOG_HAS_DLL 1
#define TAO_LOG_SERV_HAS_DLL 1
#define TAO_LOG_SKEL_HAS_DLL 1
#define TAO_LZOCOMPRESSOR_HAS_DLL 1
#define TAO_MESSAGING_HAS_DLL 1
#define TAO_NAMING_HAS_DLL 1
#define TAO_NAMING_SERV_HAS_DLL 1
#define TAO_NAMING_SKEL_HAS_DLL 1
#define TAO_NOTIFYLOG_HAS_DLL 1
#define TAO_NOTIFYLOG_SERV_HAS_DLL 1
#define TAO_NOTIFYLOG_SKEL_HAS_DLL 1
#define TAO_NOTIFY_HAS_DLL 1
#define TAO_NOTIFY_MC_EXT_HAS_DLL 1
#define TAO_NOTIFY_MC_HAS_DLL 1
#define TAO_NOTIFY_PERSIST_HAS_DLL 1
#define TAO_NOTIFY_SERV_HAS_DLL 1
#define TAO_NOTIFY_SKEL_HAS_DLL 1
#define TAO_ORT_HAS_DLL 1
#define TAO_PI_HAS_DLL 1
#define TAO_PI_SERVER_HAS_DLL 1
#define TAO_PORTABLEGROUP_HAS_DLL 1
#define TAO_PORTABLESERVER_HAS_DLL 1
#define TAO_PROPERTY_HAS_DLL 1
#define TAO_PROPERTY_SERV_HAS_DLL 1
#define TAO_PROPERTY_SKEL_HAS_DLL 1
#define TAO_PSDL_HAS_DLL 1
#define TAO_QTRESOURCE_HAS_DLL 1
#define TAO_REPLICATIONMANAGERLIB_HAS_DLL 1
#define TAO_RTCORBAEVENT_HAS_DLL 1
#define TAO_RTCORBA_HAS_DLL 1
#define TAO_RTCOSSCHEDULING_HAS_DLL 1
#define TAO_RTEC_COSEC_HAS_DLL 1
#define TAO_RTEC_PERF_HAS_DLL 1
#define TAO_RTEVENTLOG_HAS_DLL 1
#define TAO_RTEVENTLOG_SERV_HAS_DLL 1
#define TAO_RTEVENTLOG_SKEL_HAS_DLL 1
#define TAO_RTEVENT_HAS_DLL 1
#define TAO_RTEVENT_SERV_HAS_DLL 1
#define TAO_RTEVENT_SKEL_HAS_DLL 1
#define TAO_RTKOKYUEVENT_HAS_DLL 1
#define TAO_RTPORTABLESERVER_HAS_DLL 1
#define TAO_RTSCHEDEVENT_HAS_DLL 1
#define TAO_RTSCHEDULER_HAS_DLL 1
#define TAO_RTSCHED_HAS_DLL 1
#define TAO_RT_NOTIFY_HAS_DLL 1
#define TAO_SECURITY_HAS_DLL 1
#define TAO_SMARTPROXIES_HAS_DLL 1
#define TAO_SSLIOP_HAS_DLL 1
#define TAO_STRATEGIES_HAS_DLL 1
#define TAO_SVC_UTILS_HAS_DLL 1
#define TAO_TIME_HAS_DLL 1
#define TAO_TIME_SERV_HAS_DLL 1
#define TAO_TIME_SKEL_HAS_DLL 1
#define TAO_TKRESOURCE_HAS_DLL 1
#define TAO_TRADING_HAS_DLL 1
#define TAO_TRADING_SERV_HAS_DLL 1
#define TAO_TRADING_SKEL_HAS_DLL 1
#define TAO_TRANSPORT_CURRENT_HAS_DLL 1
#define TAO_TYPECODEFACTORY_HAS_DLL 1
#define TAO_UTILS_HAS_DLL 1
#define TAO_VALUETYPE_HAS_DLL 1
#define TAO_XTRESOURCE_HAS_DLL 1
#define TAO_ZIOP_HAS_DLL 1
#define TAO_ZLIBCOMPRESSOR_HAS_DLL 1

The $ACE_ROOT/include/makeinclude/platform_macros.GNU file

The simplest possible replication links to the appropriate platform-specific file in $ACE_ROOT/include/makeinclude/ for the above-listed platforms.

Contents of $ACE_ROOT/bin/MakeProjectCreator/config/default.features

n/a

AREA/CLASS/EXAMPLE AFFECTED:

The use of -ORBListenerInterfaces as documented fails. Also, the incoming pull request includes an extension new test of -ORBListenerInterfaces that, as written, failed before the changes to INET_Addr, Sock_Connect, SOCK_Dgram, etc. were applied, and now passes.

The problem effects:

Execution.

Synopsis

The configuration option -ORBListenerInterfaces does not work at all with an IPv6 address on the right side of the argument's equal sign. Additionally, undocumented behavior asserted by existing tests (use of the interface name instead of an IP address on the right side of the argument's equal sign) does not work for either IPv4 or IPv6.

Description

Documentation section 4. Connection Management and Protocol Selection addresses a configuration option -ORBPreferredInterfaces targetNetwork=localNetwork[,...]. That documentation includes such as examples as -ORBPreferredInterfaces *=*10,*=*20, referencing IPv4 addresses 192.168.1.10 and 192.168.1.20, and -ORBPreferredInterfaces *=127.0.0.1. This documentation does not mention IPv6, but the implication is that you could put full or wildcarded IPv6 addresses on either side of this equal sign in a similar manner as IPv4 addresses. We have not experimented with the full range of possibilities, but I can confirm that we are successfully using the setting -ORBPreferredInterfaces *=2001:470:c2bc:xxxx:xxxx:a062:c2fd:892e and similar (no errors, seems to work, etc., or is possibly being silently ignored).

Documentation section "4. TAO_UIPMC_Protocol_Factory" provides a similar configuration option -ORBListenerInterfaces targetNetwork=localNetwork[,...] | CopyPreferredInterfaces. It provides analog examples to -ORBPreferredInterfaces, indicating that the syntax is identical, and also details a special value CopyPreferredInterfaces to trigger copying the -ORBPreferredInterfaces value(s) to -ORBListenerInterfaces.

We first tried to use -ORBListenerInterfaces CopyPreferredInterfaces, but this resulted in BAD_PARAM errors. So instead we tried -ORBListenerInterfaces *=2001:470:c2bc:xxxx:xxxx:a062:c2fd:892e but this, too, resulted in BAD_PARAM errors, indicating I would need to dig further. Through extensive code-stepping in the debugger, I found two problem execution stacks:

ACE_SOCK_Dgram_Mcast::join -> ACE_SOCK_Dgram_Mcast::subscribe_i -> ACE_SOCK_Dgram_Mcast::open -> ACE_SOCK_Dgram_Mcast::open_i -> ACE_SOCK_Dgram::set_nic -> ACE_SOCK_Dgram::make_multicast_ifaddr6 -> ACE_OS::if_nametoindex -> ::if_nametoindex

ACE_SOCK_Dgram_Mcast::join -> ACE_SOCK_Dgram_Mcast::subscribe_i -> ACE_SOCK_Dgram::make_multicast_ifaddr6 -> ACE_OS::if_nametoindex -> ::if_nametoindex

No matter the circumstances (or use of CopyPreferredInterfaces vs the x=x syntax), at the point that this code executes:

lmreq.ipv6mr_interface = ACE_OS::if_nametoindex (ACE_TEXT_ALWAYS_CHAR (net_if));

The value of net_if is always an IPv6 address in my debugging. However, by documentation and experimentation, ::if_nametoindex does not accept IP addresses for the sole argument, and it always returns a 0 when an IP address is supplied. ::if_nametoindex requires an interface name, such as en0, eth1, utun2, etc., in order to return an interface index. So, it's not possible for this code to work. (Additional note: Existing tests in TAO/orbsvcs use options like -ORBPreferredInterfaces xxx=eth0 or -ORBPreferredInterfaces xxx=lo. CopyPreferredInterfaces results in eth0 or lo being copied, as expected, but at the above point in code execution, the value of net_if is "if=eth0" or "if=lo", not "eth0" or "lo", so that also results in if_nametoindex returning 0.)

Furthermore, inspecting the make_multicast_ifaddr6 implementation in more detail, this is also broken on Windows, where the value of net_if is also always an IPv6 address, because it's comparing net_if to the interface's adapter name or friendly name:

                       (ACE_OS::strcmp (ACE_TEXT_ALWAYS_CHAR (net_if), pAddrs->AdapterName) == 0
                        || ACE_OS::strcmp (ACE_TEXT_ALWAYS_WCHAR (net_if), pAddrs->FriendlyName) == 0)))

Comments throughout ACE/TAO code suggest that net_if should be an interface name at this point, not an IP address, but I could find no code anywhere that actually performed this conversion. net_if is taken from the right-hand side of the equal sign in -ORBListenerInterfaces and passed unchanged all the way down the stack.

My first instinct was to go somewhere up the stack and convert net_if from an IP address to an interface name, but it quickly became apparent this was not the correct choice. Doing so breaks make_multicast_ifaddr (the IPv4 branch of code). The difference lies in lmreq.ipv6mr_interface vs lmreq.imr_interface.s_addr. The kernel's IPv4 stack expects a source IP address (lmreq.imr_interface.s_addr), but the kernel's IPv6 stack expects a source interface unsigned index (lmreq.ipv6mr_interface)—inconsistent, but unchangeable. So make_multicast_ifaddr is implemented (*mostly) correctly, and net_if needs to be an IP address in order for it to be implemented correctly. Thus, make_multicast_ifaddr6 is what needed changes—specifically, to convert net_if from an IPv6 address into an interface name, or something similar. (*Additional note: make_multicast_ifaddr has the same undocumented behavior of accepting an interface name [and converting it to an IP address instead of an interface index], but it, too, suffers from the "if=eth0" / "if=lo" bug.)

The approach I took might not be the approach you want to take to fix this, but it's the patch we applied to our ACE/TAO copy that got this working. Notably:

• There could be a simpler fix—I took the route of assuming that net_if could be either an IPv6 address or an interface name, and checked for both, because that's the existing behavior and also existing tests use this behavior, but maybe that wasn't necessary and this behavior can be removed and the tests changed.
• There could be other places that net_if is incorrectly assumed to always be an IPv6 interface name, that I just didn't find because our use cases aren't hitting those code paths.
• There could be similar problems with -ORBPreferredInterfaces that either cause it to silently be ignored or that we aren't encountering because our uses cases aren't hitting those code paths.
• There could be other, system IPv6 deficiencies that we just haven't found yet (TAO appears to largely be working with IPv6 for us).

Repeat by

The incoming pull request has a new test $TAO_ROOT/orbsvcs/tests/Miop/McastListenerInterfaces. This test contains the code and configuration necessary to replicate the problem. It was implemented first, and failed. Then our patch was applied, and the test passed. NOTE In order for this test to compile and run, $TAO_ROOT/TAO_ACE.mwc has to be edited to comment out or remove the line orbsvcs/tests in the exclude block. It seems that all orbsvcs tests are currently disabled in master. I did not include this change in my pull request, because it seems those tests were disabled for a reason, which I won't second-guess. I know, for example, that most of the tests in $TAO_ROOT/orbsvcs/tests/Miop/ fail on a clean master.

Sample fix/ workaround

There is no workaround. -ORBListenerInterfaces does not work without changes. The incoming pull request is a fix that works, but you may prefer to fix it another way. Furthermore, on some macOS machines (I'm unable to find an explanation for why some and not others, but I can confirm that the macOS version is not a factor), IPv6 multicast traffic does not work at all without the -ORBListenerInterfaces config option, because macOS cannot determine which interface to use automatically. This could be an obscure macOS bug related to specific hardware versions.