Multiple Instances

[noelhibbard]

@mikebrady , I am currently running multiple instances of SPS-AP2 in Docker containers using PulseAudio but I noticed in the What's New post you said that NQPTP has some changes that allow multiple instances to work on a single machine. You also said that clients get confused about multiple instances on the same machine. I gave this a spin and it does seem like the clients don't understand and one instance overwrites the last. So basically this is unusable, or is there a work around I'm missing?

Looks like for now I will still need to use containers to run multiple instances.

[mikebrady]

Hi Noel. Yes indeed, it seems that clients, including iOS and the Mac Music app, get confused by multiple instances on the same IP number. Unfortunately, it seems to be a dead end at the moment.

[noelhibbard]

Okay, thanks for the confirmation.

[YutongGu]

I wanted to revive this discussion and shed some new light with my recent findings. It appears that Airplay2 clients are not being confused by instances with the same IP address, rather by instances that share the same 'DeviceID' in their mDNS advertisement.

I tested this by creating macvlan interfaces on my Linux machine and assigning unique IP addresses to each of them from my router's DHCP server. I then ran an instance of Shairport-Sync (with Airplay2 enabled) on each interface by providing them with different config files with "general.interface" set to the appropriate macvlan interface.

I validated that each shairport-sync instance was advertising from the appropriate IP address by running tcpdump of mdns traffic on a separate machine.

With that, I checked on my iPhone 13 Max Pro (iOS 15) for both instances and it showed only one appearing still in my Airplay menu.

After failed test, I tried a different approach. I tried to modify the 'DeviceID' field advertised in the mDNS packet for the Shairport-Sync instance. To do this I overwrote the "general.airplay_device_id" field in the config file for each instance to be unique and I was finally able to see both instances appear in my airplay menu on my iPhone.

My follow up questions are, would there by any unintended side effects of defining the airplay_device_id to be an arbitrary value? Would it affect streaming/controls at all? I will investigate further and provide my findings here. I'm also happy to provide more details as to how I tested this so people can reproduce this experiment.

[noelhibbard]

My guess is the USB bus on the RPi can't handle the traffic. I've never tried multiple instances on an RPi.

[OptimoSupreme]

Oh man @YutongGu this is awesome. How did you even discover the airplay_device_id_offset variable? In any case, I was able to get my pi running a few instances of shairport! I wonder if I can find a way to include some information about multi instance sps in the readme or something. I wrote a small instruction doc on how I did it. Anyways, thanks man this was kicking my butt... I though it was a networking problem for a long time.

Multiple Instance Pi 3b Setup.md

[noelhibbard]

I wrote a small instruction doc on how I did it. Anyways, thanks man this was kicking my butt... I though it was a networking problem for a long time.

Multiple Instance Pi 3b Setup.md

So are you using an older build of NQPTP to get this to work?

[OptimoSupreme]

I didn’t compile SPS, I ended up installing via dnf so I’m using whatever it ships with.

[YutongGu]

@OptimoSupreme Glad you were able to recreate it with your own setup! Finding that airplay_device_id_offset took a bit of code spelunking and documentation perusing 😅 Unfortunately, it seems that multiple instances may not work reliably with newer versions of SPS as @noelhibbard reported. So keep using what you're using now if it works and hopefully support will be added back soon.

[mikebrady]

Thanks for all this. I tried to explore this some time back and the general.airplay_device_id and airplay_device_id_offset fields were added at that time to enable separate instances of Shairport Sync (SPS) to have stable but different Device IDs. NQPTP was also extensively modified to handle multiple Shairport Sync instances on the same system.

Experiments seemed to show that AirPlay clients routinely confused two endpoints on the same IP even when they had different IDs. This happened with macOS and iOS clients. Of course I could be wrong.

The thing is that since that time, NQPTP's ability to handle more than one client has been removed (greatly simplifying NQPTP), so the situation @noelhibbard describes now could easily be caused by one SPS instance turning off NQPTP, which would have the effect of turning it off for all instances. One could go back to previous versions of SPS and NQPTP, before the multi-client facility was removed from NQPTP, but there have been quite a few bug fixes since then.

You might have seen that an AirPlay 2 version of SPS has just been released, so it will take some time to enable NQPTP to serve multiple instances of SPS again. SPS itself would have to be modified too, but that's much simpler.

[YutongGu]

Thank you @noelhibbard for digging into this and thank you @mikebrady for chiming in on this!

Is there a timeline we can expect adding support for multiple SPS instances? Being able to launch multiple instances of SPS with Airplay2 without depending on Docker would simplify things drastically. It would also be nice to be able to provide device ID as a parameter on invocation from the command line, rather than creating separate config files for each instance.

Understanding that you're probably very swamped with tons of other request, I'd also be happy if you could share the commit hash of the version of NQPTP that supported multiple instances of SPS (and perhaps what bugs to expect).

[YutongGu]

Update: I was finally able to test this out for myself and. I just so happened to be running an older version of NQPTP and shairport-sync AP2 that work well together and I was able to play multiple instances without the side effects @noelhibbard mentioned.

NQPTP:
branch - main
commit - c91a2bb209646c3253eed1d32d56614902380fda

Shairport-Sync:
branch - development
commit - 250704b

[noelhibbard]

Did you happen to test all these scenarios?:

• transport controls in buffered and realtime
• ability to add/remove endpoints in buffered and realtime

[YutongGu]

Sorry I'm not familiar with the terminology. I'm not sure what you mean by buffered and realtime. Could you clarify?

Also by adding/removing endpoints do you mean selecting and deselecting Airplay2 instances from the Airplay menu?

[YutongGu]

So here's what I've tested. I would launch two instances of shairport-sync AP2 with different airplay device ids. We'll call them Input-A and Input-B. I would then start playing from Spotify. With music playing, I would connect to one or both instances and observe how volume control and playback control behaves. Here were my observations:

• I could play to both Input-A and Input-B simultaneously and control their playback/volume independently
• I could disconnect from Input-A without any affect on Input-B's playback and vice versa

Some glitches were that I would sometimes see the two Airplay2 inputs appear under the same name with a dropdown showing two additional Airplay inputs (see attached picture). I haven't been able to reliably reproduce it and it seems to resolve itself most of the time.

[YutongGu]

Update: Apparently this is an issue with my iOS. I've since upgraded to iOS 16 and this no longer occurs.

[CDaxi]

If multiple instances on the same ip address is an issue, could it be possible to bind an instance to a specific ip address and allocate multiple ip addresses for example on a LXC container in proxmox?

Allocation multiple networks which gets its own ip address and mac address is fairly easy at Proxmox, but I see no option to specifiy an ip address to shairport-sync.
Currently, I think, the service will bind to the specified port on all ip addresses on the machine, right?

I have the following setup in my mind:

• One LXC container on proxmox
• two 7.1 sound cards
• multiple IP addresses allocated
• bind each instance of shairplay-sync to one of them
• additionally mopidy and snapserver / snapclient should run on the machine.
• piping the audio from shairplay or mopidy to snapserver is already possible.
• running snap clients for a multiroom audio server
• additional external sound devices will be possible by connecting more snapclients to the snapserver instance.

If this would be possible, a very nice multiroom audio setup can be created :-)

Thanks a lot.

Kind regards,
CDaxi

[noelhibbard]

@CDaxi , I’ve had success with Proxmox, LXC and Docker. Getting complex ALSA configs to work in containers doesn’t work well though. I had to switch to Pulse to get my setup to work correctly. I prefer doing it in docker because I can manage all my instances in one docker-compose file. I’ve documented my setup here: https://github.com/noelhibbard/sharport-sync-docker I also run a Snapchat server and multiple instances of snapclient and librespot but I do that directly on the host.

[mikebrady]

@noelhibbard, can you say more about the difficulties in working with the ALSA subsystem in Docker? What seems to be the problem? Thanks.

[noelhibbard]

My ALSA config busts up a few 7.1 channel cards into multiple outputs using dmix. This config has worked for years for multiple instances of SPS-AP1 on bare metal but in Docker it will only let me playback to one of these dmix outputs at a time. If another instance attempts to use another dmix output that is tied to the same physical card, it says the device is busy. I don't think it's an issue with SPS, I think it's just some strange issue with how Docker interacts with ALSA. LXC containers seem to have the same limitation. My guess is there is some sort of locking situation with how Docker mounts /dev/snd into the containers.

[mikebrady]

Okay, thanks -- I guess you are right.

[clumsyCoder00]

Thanks for documenting your process. I'm running a similar setup using ALSA, and running into similar issues.

[noelhibbard]

Thanks for documenting your process. I'm running a similar setup using ALSA, and running into similar issues.

Mike got me motivated to try to get ALSA working properly in docker and I spent a few hours working at it some more and think the issue has to do with IPC. I found some settings related to how docker handles IPC but haven't had a chance to do more testing. I need to make a test rig to make troubleshooting easier.

[clumsyCoder00]

A test rig is a good idea! I used your docker setup earlier, and was able to connect to a single instance and even get audio through to the host machine, but I can't disable the pulseaudio service on the host because it's being used for my active shiarport host. I'm also using Ubuntu Desktop so it's a bit of a different environment than yours. I don't have a strong understanding about the network bridge you were using in your example, and have a bit to learn in that area as well.

[YutongGu]

I'm currently using my method of running multiple instances of Airplay2 which involves manually specifying the device id and using an older version of nqptp/shairport-sync.

It all works but I'm running into an issue where things start to get slow after I get to my 2nd or 3rd simultaneous Airplay2 input. It takes a while to disconnect from the Airplay2 instances and i have to wait until the busy spinner on the airplay screen goes away (presumably because something timed out) before I'm able to do anything again. After that, things remain slow to connect.

Has anyone seen anything like this when running multiple Airplay2 instances? Could it have something to do with being on an older version of shaiport-sync/nqptp?

[YutongGu]

When I disconnect from multiple instances of shairport-sync by selecting 'iPhone' (rather than deselecting each input one by one), I get a pinwheel spinning indefinitely. When done from the control center, it renders Airplay selection inoperable.

@noelhibbard Has this ever come up in your testing with your docker instances?

[clumsyCoder00]

I've gotten multiple instances of shairport-sync to run in a docker-compose stack on a virtual network managed by docker. The host in this case is Ubuntu 22.04 LTS. I've posted my config in my repository for reference. Thanks to everyone for their efforts!

[wildekek]

Beware that running multiple instances over WiFi does not work with docker and Macvlan. Ask me how I found out :(
I'm considering running pimox on my Pi to resolve this issue.

[shifttymike]

So I thought I was clever and did the following:

• Created a new network namespace with a dedicated veth, with the other end of the veth in a bridge with the host's main interface. So from the router's perspective, there are 2 IPs with 2 associated MACs
• Cloned the source for nqptp and shairport-sync and changed the shared memory interface path to /nqptp2 Confirmed that /dev/shm contains both
• Compiled and ran both applications within the network namespace with separate config (different alsa device and target name)

Both AirPlay targets A and B show up. A being the original one running per usual, B being the new one in the namespace. Success? I select "B" to stream to from my phone, "B" immediately disappears and "A" is now selected and receiving the stream without me ever selecting it. There's other weird behaviour, but in short it always plays through "A"

How is it that with a completely separate network stack, IP, MAC address and SMI path, devices are still getting confused between the two targets? What am I missing that's still common between the two? I'm determined to make this work without having to spin up a VM.

[mikebrady]

Thanks for the update. From experience, I'm inclined to agree with your diagnosis. Of course this must be speculation, but it occurs to me that some parts of Apple's AirPlay 2 client implementations use the airplay_device_id but other parts rely on the IP numbers! Avahi's behaviour doesn't help, but it may (?) be that it is working this way to conform with the design of the Bonjour protocol.

[acmay]

@shifttymike Can you do tcpdump/wireshark packet captures?

Are you doing IPv4 or IPv6?
What Airplay clients are you using currently?

Can you run avahi inside the namespace too?

Here is a hardcoded hack to change avahi registration 0b3029e

To add in some different host names I used this command.
"avahi-publish -a piamp2_henry.local -R 192.168.1.241"

I have been running IPv6 recently without seeing the same problems with IPv4.

[acmay]

I switched back to my raspberry pi setup and I was able to see the confusion problem again.
I have no idea why I didn't see the same problem on my full desktop setup running Manjaro.
I can try to see if I can find out how it is different.

On the raspberry pi setup it seems that by doing a manual publish of 2 different hostnames then doing the register just on the hostname gets past the confusion problems. It can be different IP addresses too, but it seemed to work with the same IP.

The hostname publish commands.

avahi-publish -a house.local -R 192.168.1.116
avahi-publish -a henry.local -R 192.168.1.116

This is the change to publish just on the hostname that matches the "name" config option (plus the clock changes)

--- a/mdns_avahi.c
+++ b/mdns_avahi.c
@@ -250,6 +250,13 @@ static void register_service(AvahiClient *c) {
       return;

     int ret = 0;
+    char *host = NULL;
+    char full_host[256];
+       debug(1, "here\n");
+       debug(1, "here %s\n", config.service_name);
+    snprintf( full_host, sizeof(full_host), "%s.local", config.service_name);
+    host = full_host;
+       //debug(1, "here %s\n", host);

     AvahiIfIndex selected_interface;
     if (config.interface != NULL)
@@ -258,7 +265,7 @@ static void register_service(AvahiClient *c) {
       selected_interface = AVAHI_IF_UNSPEC;
     if (ap2_text_record_string_list) {
       ret = avahi_entry_group_add_service_strlst(group, selected_interface, AVAHI_PROTO_UNSPEC, 0,
-                                                 ap2_service_name, config.regtype2, NULL, NULL,
+                                                 ap2_service_name, config.regtype2, NULL, host,
                                                  port, ap2_text_record_string_list);
       if (ret == AVAHI_ERR_COLLISION) {
         die("Error: AirPlay 2 name \"%s\" is already in use.", ap2_service_name);
@@ -266,7 +273,7 @@ static void register_service(AvahiClient *c) {
     }
     if ((ret == 0) && (text_record_string_list)) {
       ret = avahi_entry_group_add_service_strlst(group, selected_interface, AVAHI_PROTO_UNSPEC, 0,
-                                                 service_name, config.regtype, NULL, NULL, port,
+                                                 service_name, config.regtype, NULL, host, port,
                                                  text_record_string_list);
       if (ret == AVAHI_ERR_COLLISION) {
         die("Error: AirPlay 1 name \"%s\" is already in use.", service_name);

Then mdns records.

Answers
B827EB3C0F30@henry._raop._tcp.local: type SRV, class IN, cache flush, priority 0, weight 0, port 7001, target henry.local
henry.local: type A, class IN, cache flush, addr 192.168.1.116
house._airplay._tcp.local: type SRV, class IN, cache flush, priority 0, weight 0, port 7000, target house.local
house.local: type A, class IN, cache flush, addr 192.168.1.116
henry._airplay._tcp.local: type SRV, class IN, cache flush, priority 0, weight 0, port 7001, target henry.local
B827EB3C0F17@house._raop._tcp.local: type SRV, class IN, cache flush, priority 0, weight 0, port 7000, target house.local

[shifttymike]

Thanks @acmay! Appreciate the detailed output.
To answer your questions, I'm running dual-stack IPv4/6 and clients are iPhones/iPads. All Apple stuff.

The issue seems to be that I'm only using a netns, everything else (process/filesystem) is common. Avahi is exposed via /run/avahi-daemon/socket so the API "escapes" the netns through the socket.
Since the hostname is the same for both instances of shairport-sync, it's published with the same hostname/ipv4/ipv6 tuple for both instances and when it connects to the service, it's always going to get the instance in the default netns. So the behaviour makes sense and your patch fixes it by changing the hostname and publishing the alternative address.

That being said, the machine I want to run multiple instances on is already running Proxmox making LXC containers trivial to deploy, so I'm going to go that route instead (pretty much the same as Docker). The main reason is that I'd prefer not to patch Shairport if I can help it, will make upgrading easier and the resource cost is close to nil in the LXC (16MB RAM usage while streaming).

[acmay]

What iOS versions? I am at iOS 18.0.1 and Sequoia 15.0.1.

I am also wanting to run this on a raspberry pi, so I am trying harder to avoid containers.

This issue has been around for a few years so I would guess it is back to iOS 15 where issues were first seen.

I have not had great luck reproducing the problem even without the patch and a single IP now, so it is hard to say this change is even needed. Maybe it is just enough to be on things newer than iOS15.

My next guess is maybe the problem shows up when the 2 different instances trigger different mDNS packets at startup. After the info gets combined into one packet (like the one I snipped above) it works fine. But during the time when the are 2 different mDNS packets the clients may get confused.

Either way can you try with no patches a single IP, but different ports/device id's. Do your recent iOS versions still get confused?
If so is it just after startup or does it work better after waiting?