Currently when using switch_to_blog, it doesn't also switch the network (site) id as well. Details of this issue can be found in a core ticket #25293. The issue is related to not wanting to add a database queries to the switching process. This is a valid problem. In core patch #40513, the switch_to_blog function will allow for WP_Site to be passed. This WP_Site will then be passed to 'switch_blog' action. This is a good for multi network. With the WP_Site we also get a property of network_id. A action could be added like this

function switch_blog_and_network( $new_blog, $prev_blog_id, $site_object ){
       if ( $site_object instanceof WP_Site && get_current_network_id() != $site_object->site_id ) {
              switch_to_network( $site_object->site_id );
       } else {
             restore_current_network();
       }
}
add_action( 'switch_blog', 'switch_blog_and_network', 10, 3 ); 

This would also switch network if different from current network.