Crazyflies loosing poses for swarm sizes greater then 6 drones

[darshit-desai]

Hi @whoenig,

I have been testing the crazyswarm API in a new lab, The lab setup is given as below:

Tracking: Vicon Vantage V16 Cameras working at default 100 FPS. No. of cameras covering the space 40
Area of the space: 80’ by 60’ space with a minimum 30’ ceiling
The lab is RF shielded ( 90 MHz-18 GHz )

Connection with the computer running Crazyswarm: Connected via ethernet cable with the Vicon Computer directly over static IP with separate IP domain

Computer specs running the Crazyswarm: Razer Lambda X Tensorbook running native Ubuntu 22.04

Drone swarm configuration:
3 Radios on 3 channels over 80, 90 and 100
2 radios connected via USB splitter and 1 radio directly to the Computer

Issue description: The hoverswarm launch script runs and initializes radio and crazyswarm server connection with the vicon and drones, As soon as I start the figure8_csv.py script the drones start loosing pose. Although they takeoff pretty well, but every drone after takeoff starts loosing pose and lands after some time.

What we have debugged?

1. Vicon, we have recalibrated vicon multiple times and also checked the data delivery latency which is well within the specifications of crazyswarm (After all overheads the data is delivered to the Linux computer running crazyswarm at 70 Hz)
2. Tried scaling down the drones from 18--->12 but that also doesn't work
3. Tried using bigger markers and smoother higher quality markers but that also doesn't seem to resolve the issue

What we think might be wrong?

Last week me and @ewinge1 had a zoom call to debug the same issue and the only bottleneck we found was the communication between radios and drones, Eric specifically pointed out that while rebooting using chooser.py script some of the drones timed out and didn't reboot. This could be one of the bottle necks. Since the lab is RF Shielded the radio interference theory could be ruled out. The only other thing we could think of was something in the server script delaying the communication

Comparative testing of the above theory with Crazyswarm 2: (CSW2)

To test the radio failure theory, we tested the same 12 drone swarm on CSW2, the drones did takeoff and some executed figure 8 while some didn't (which is an entirely separate and open issue on CSW2 repo) but none of the drones reported Pose Lost errors. Eric said that since CSW2 uses Safelink protocol the poses are guaranteed to be delivered even though it takes time, while in case of CSW1 there is a timeout which happens, which basically means the radio gives up on the drone.

Contradictions:
I tested a 24 drone swarm using CSW1 in a smaller lab with 12 cameras and it worked just fine (i.e. the figure8 script), plus that lab is not RF shielded and is infact much worser environment for 2.4G devices to get tested.

If you could give me some pointers about this issue it would be greatly helpful

[whoenig]

Thanks for the report including the comparison with Crazyswarm2. A few notes

• If Crazyswarm2 works apart from some drones missing unicast commands, you can send those multiple times. This is, as far as I know, what Eric has been doing. This issue is of course very high on our list, although it might require firmware changes, too.
• For the communication, can you run comCheck (part of crazyflie_tools)? This can measure the latency, packet drop rates and similar.
• What tracking mode are you using? Single-marker or multiple markers? libobjecttracker or VICON Tracker?
• When you observe "losing pose", is this related to error messages on the terminal, or the drones showing erratic behavior?

[Eric-Ewing]

If Crazyswarm2 works apart from some drones missing unicast commands, you can send those multiple times. Can confirm that is what we've done here. Darhsit's communication situation is worse than ours for some reason and that has made debugging a bit harder as I can't recreate the problems locally. The biggest red flag to me is the timeouts when calling reboot or sysoff from chooser.py didn't seem to be tied to specific robots or channels. The fact that the timeouts occur without the server running (and thus without pose packets) made me much more concerned about radios than about tracking. I believe he has also seen broadcast messages being missed by some drones in CS2 and not just unicast messages, which we've never had issues with at Brown.

[whoenig]

Broadcasts are automatically repeated (and a user can configure how often), so technically this should be fine for bad connections. But the safelink bug together with poor connections might render CS2 inoperable until this firmware bug gets fixed.

[Eric-Ewing]

I think even with 30 repeats (twice what we use), they were still missing some commands. It's just an indication of how much worse the connection is than I have locally.

[darshit-desai]

Again it is worth mentioning that I tested the same overkill broadcasts and unicasts configuration in a smaller lab with 24 drones and it had some moderate level of success. I boosted the upload trajectory and start trajectory commands by 12 times. (While others were repeated 3 times)

[darshit-desai]

For the questions

• I am running the swarm server using libobjecttracker with single markers.
• The losing pose is a specific warning which appears as soon as the python script starts running, (This warning doesn't come when the hover_swarm launch file is launched). Although during connecting to the drones (i.e., after pressing enter for the launch file launch command), there are multiple warnings of poor link quality until it is able to connect with all the drones. Below is a snapshot of the same pose lost warnings, (This is basically reflected in real world where the drones after takeoff immediately start to drift)
  

For comCheck is there a specific command I need to run after the server is launched. I want to clarify the overall sequence of running this diagnostics, is it rosrun crazyflie_tools comCheck.py or something

Yes, I had a moderate amount of success with explicit unicasts, and me and Eric found that there are explicitly some commands which even after doing unicasts are not reaching the drone, those are either the upload_trajectory or start_trajectory commands. (Not able to narrow down beyond that)

[whoenig]

It should be rosrun crazyflie_tools comCheck --uri <URI> (and no server running in parallel).

If the link quality is very poor, there seems to be a bug in safelink that can cause infinite communication, which might have been what you are seeing with Crazyswarm2. It's documented in bitcraze/crazyflie-firmware#703, but sadly hasn't seen much progress.

[darshit-desai]

If the link quality is very poor, there seems to be a bug in safelink that can cause infinite communication, which might have been what you are seeing with Crazyswarm2. It's documented in bitcraze/crazyflie-firmware#703, but sadly hasn't seen much progress.

Actually I highlighted this for CSW1, the link quality errors come with CSW1 launch server (and not with CSW2)

[darshit-desai]

Was the purpose of getting the above diagnostics data to ask Bitcraze RF engineers for a solution?

[whoenig]

It was mostly to rule out that the problems you are seeing are related to Crazyswarm (1 or 2), but rather an issue with the specific environment. While we can do a few more things in Crazyswarm2 to support poor communication better, some of your message outages were so significant that other changes (such as an external antenna) are likely to be needed.

[darshit-desai]

Can we contact them anyways? I think that this is not the last time anyone is testing it in a RF shielded environment and there must be solutions like you suggested, mounting antennas on the drones, similar might be available for the radios too, maybe a different type of antenna or something. And if it is true that the radios rely on reflected waves (and the lab walls are absorbing those) there might be stopgap solutions like mounting radios on the drones.

[whoenig]

Sure, just post on https://github.com/orgs/bitcraze/discussions, and also tag me and include a link to the discussion here.

[darshit-desai]

Thanks @whoenig just mentioned this issue in this thread https://github.com/orgs/bitcraze/discussions/1016#discussion-5806395 on bitcraze discussions