Failed to load nodelet and parameters with dev/external trigger branch

[oysteinvolden]

System Description
Camera Name and Model: BFS-PGE-13Y3C-C
Operating System: Ubuntu 18.04
Spinnaker Version: 2.0.0.147
ROS Version: Melodic

Computer details
Nvidia Jetson Xavier Development Kit
Processor: ARM 64 bit (v8.2)
RAM: 32 GB

Multiple Cameras Only
How is the triggering setup? all slaves (external trigger)

For Ethernet cameras:
How busy is the network, does the system work in grayscale, at a lower resoultion (binning) and framerate? Network is working, both at lower and higher resolution (binning) and framerate.

Do the cameras work with SpinView? Yes. In particular, it also works with external trigger (1 fps) in Spinview. We can clearly see that the cameras are triggered at the same time. Both cameras set trigger mode on as well as enable line3 (pin 1) and readOut. On the hardware side, both cameras are setup as slaves using the figure under "GPIO Connections for Master/Slave config" in the README. We use an external trigger (https://www.ntnutto.no/prosjekter-items/sentiboard-2/) which act as the master to trigger both cameras. Again, the spinview application confirm that the hardware are connected correctly.

Describe the bug
I am not able to launch the camera driver with external trigger where two cameras are supposed to be slaves. We have tried both the external_trigger branch and the dev branch without luck. We followed issue #89 to compile the driver with nvidia Xavier 18.04 successfully. We have correctly setup the external_trigging_params.yaml file with correct serial numbers and updated the acquisition.launch to find this file and also set external_trigging to true. Also binning are correct. We have also tried other versions of Spinnaker (1.27.0.48 and 2.2.0.48) without luck.

Screenshots
First we tried using the dev branch, where the driver fail to load the nodelet as seen in the screenshot. We can also see that camera IDs and camera aliases are not found. This yields for all the cases we have tried below.

Then, we tried the same branch but now using rosrun instead of roslaunch. As seen in the screenshot below, the driver detects the cameras but segfaults after acquisition. In addition, the parameters in the acquisition.launch file are not set.

Furthermore, we tried roslaunch with external trigging branch and we get the same failure as with dev branch (using roslaunch).

Then, in the same manner, we tried external triggering branch with rosrun and it segfault also here although the cameras are connected. Also note that we have also tried to convert from hexa to integer serial number as shown in #84 with luck (not shown in the screenshot), but it will still fail.

At last, we can see that the master branch works as it should but using only test_params.yaml and a master/slave setup which is not what we want.

So, it seems to be error when loading the nodelet using roslaunch (with external trigging/dev branch). And it seems to be some error loading the parameters using rosrun (and roslaunch). Do you have some ideas what may be wrong?

[ghost]

@oysteinvolden thanks for the detailed bug report. We have successfully used the external trigger with an x64 laptop. Could you try running your setup with a laptop just to isolate whether it is a driver issue or a architecture / system / compiler (C++ version) / hardware / memory issue. @jpsnir have you seen segfaults with external trigger?

[oysteinvolden]

I tried the dev branch with x86 laptop (Ubuntu 18.04 / ROS melodic) and it actually works after commenting a couple of lines, mostly ROS ASSERTION. This was verified by launching two rqt_image_view so we can easily see that the cameras are triggered at 1 fps. I used Spinnaker 2.0.0.147 at the x86 laptop as well. See screenshot below.

I retried the dev branch on the Jetson Xavier using the exact same configuration. I got however the same result as before. I looked into capture.cpp and it seems like the parameters are not set as earlier mentioned. In particular, everywhere nh_pvt_getParam() are used in the code. Hence, relevant parameters such as external trigging are not set correctly (default value) which again will propagate new errors using external trigging setup.

Do you have some ideas why the parameters from the launch file are not detected in capture.cpp? This yields for both external trigging and dev branch, but not master branch using the Jetson Xavier.

Btw, both x86 laptop and x64 (jetson) use gcc 7.5.0 compiler.

[oysteinvolden]

Finally it works now. The main issue seems to be that "cam_aliases" and "cam_ids" are not found by the getParam() function at line 321 and 329 in capture.cpp. I ended up commenting line 321-339 in capture.cpp and hardcode the cam ids / cam aliases below line 339 in capture.cpp. See screenshot of the capture.cpp file below.

It seems like it is only the parameters from the external_trigger_params.yaml file that are not detected (without the modification above), while all the other parameters in acquisition.launch will be detected when cam ids / cam aliases are found.

[ghost]

@oysteinvolden roslaunch loads the config from the yaml file. Are you sure there are no issue with the formatting in the yaml files. You can try parsing the file using python or another yaml file parser to be sure. Can you share your config file?
Also, a little nitpick, it better to copy pase code / terminal output instead of putting screenshots. You can use code blocks <> so that it formats and scrolls correctly. That way you can send the full information instead of just the last screen.
Glad, that you were able to get it to work.

[oysteinvolden]

The external_trigger_params.yaml file can be seen below.

cam_ids:
- 19176570
- 19176568
cam_aliases:
- cam0
- cam1
skip: 20
delay: 1.0

flip_horizontal:
- false

flip_vertical:
- false

#Assign all the follwing via launch file to prevent confusion and conflict

#save_path: ~/projects/data
#save_type: .bmp #binary or .tiff or .bmp
#binning: 1 # going from 2 to 1 requires cameras to be unplugged and replugged
#color: false
#frames: 50
#soft_framerate: 4 # this frame rate reflects to the software frame rate set using ros::rate
#exp: 997
#to_ros: true  #When to_ros is not selected, but live is selected, pressing 'space' exports single image to ROS

I cannot see what is wrong with the file, but I can try to use a yaml file parser. Do you see any errors? The serial numbers are correct. Here is a screenshot of the file as well.

[ghost]

Yes I agree, I don't see any obvious issue with the file. It is just surprising that the code works fine on your x86 system and not on the Jetson. Just to confirm, are using roslaunch or rosrun to launch the nodes?
Also can you paste the actual output when the node fails to read the params. You can use ``` ``` for multiline code blocks.

[oysteinvolden]

In my final setup with external trigger in dev branch I used roslaunch to launch the node. I don't have the Nvidia device in front of me now, but they are pretty much summarized in the screenshots above. If we focus on the dev brench (which works now), you can see the actual output in the two upper screenshots of the initial issue (the first for roslaunch and the second for rosrun). Hence, we get the "failed to load nodelet" failure when the node fails to read external_trigger_params.yaml using roslaunch and a segfault after "*** ACQUISITION ***" when using rosrun.

[ghost]

Reading the yaml is done outside the code. Roslaunch reads there files and loads the parameters to the ROS parameter server. Our node reads it from the parameter server. Does your acquisition.launch point to the correct .yaml file? You can also check if it is read into the ros parameter server by running rosparam list, you might have to start a separate roscore before running the launch file so that the ros master doesn't close when the node seg faults.