Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault #2

Open
FatihSarigol opened this issue Nov 12, 2021 · 8 comments
Open

Segmentation fault #2

FatihSarigol opened this issue Nov 12, 2021 · 8 comments
Assignees

Comments

@FatihSarigol
Copy link

Hello,
We've been receiving segmentation fault with the cloud analysis and we guess it is because the MPI is not set up correctly during installation but we tried several things but nothing helped. Do you have any suggestions? The collinear analysis works fine.
Thanks!

@FatihSarigol
Copy link
Author

FatihSarigol commented Nov 12, 2021

I just installed it again, using GCC 7.4 this time, instead of 10.2, and did not receive the warning messages that I received before, so probably that was the reason.

@FatihSarigol
Copy link
Author

FatihSarigol commented Nov 13, 2021

But yet again we get another error with SyntenicCloud:


size of pack(buffer) statement = 17
i-adhore: /i-adhore-3.0.01/src/SynthenicCloud.cpp:320: int SynthenicCloud::pack(char*) const: Assertion `bufSize==getPackSize()' failed.
[vlogin2:00602] *** Process received signal ***
[vlogin2:00602] Signal: Aborted (6)
[vlogin2:00602] Signal code:  (-6)
[vlogin2:00602] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2b2e6c579630]
[vlogin2:00602] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b2e6d4f8387]
[vlogin2:00602] [ 2] /lib64/libc.so.6(abort+0x148)[0x2b2e6d4f9a78]
[vlogin2:00602] [ 3] /lib64/libc.so.6(+0x2f1a6)[0x2b2e6d4f11a6]
[vlogin2:00602] [ 4] /lib64/libc.so.6(+0x2f252)[0x2b2e6d4f1252]
[vlogin2:00602] [ 5] /bin/i-adhore(_ZNK14SynthenicCloud4packEPc+0x696)[0x428a86]
[vlogin2:00602] [ 6] /bin/i-adhore(_ZN14SynthenicCloud19packSynthenicCloudsERKSt6vectorIPS_SaIS1_EEPc+0x38)[0x428e28]
[vlogin2:00602] [ 7] /bin/i-adhore(_ZN7DataSet23parallelLevel2ADHoReDynEv+0x278)[0x41fbc8]
[vlogin2:00602] [ 8] /bin/i-adhore(main+0x29e)[0x41a19e]
[vlogin2:00602] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b2e6d4e4555]
[vlogin2:00602] [10] /bin/i-adhore[0x41a6cf]
[vlogin2:00602] *** End of error message ***
Aborted

@bedroesb
Copy link
Member

bedroesb commented Nov 15, 2021

@FatihSarigol I am glad you found a solution for the first problem. Did you already try running I-adhore using our container? There is one available on dockerhub which can also be used with singularity.

I would also like to know more info about you OS, version of cmake and which command you are running with which type of data.

@FatihSarigol
Copy link
Author

Thank you for your reply Bert,
I also thought about trying docker but we don't have admin access on our HPCC.
Do you happen to have a list of versions of things that you used when it worked for you? Maybe we can already try to bring together the same things on our side if possible and it already works I hope.

Here are the versions in our system:

OS: CentOS Linux 7
cmake version: 2.8.12.2
Open MPI version: 3.1.3
To install this one, the version of gcc that I used: gcc/7.4.0

We used the dataset II. ini , just added three lines to the config file:

(cluster_type=hybrid
cloud_gap_size=20
cloud_cluster_gap=25)

Here is the complete report with parameters:


This is i-ADHoRe v3.0.
Copyright (c) 2002-2010, Flanders Interuniversity Institute for Biotechnology, VIB.
Algorithm designed by Klaas Vandepoele, Cedric Simillion, Jan Fostier, Dieter De Witte,
Koen Janssens, Sebastian Proost, Yvan Saeys and Yves Van de Peer.

Process 1/1 is alive on compute node.

WARNING: Maximum allowed number of gaps in the alignment not specified.  Setting to cluster_gap.
WARNING: Tandem gap size not correct in settings file. Using default (gap_size / 2)


************* i-ADHoRe parameters *************
        Number of genelists = 420
        Blast table = iADHoRe_PLAZA.table
        Output path = output/
        Gap size = 30
        Cluster gap size = 35
        Cloud gap size = 20
        Cloud cluster gap size = 25
        Max gaps in alignment = 35
        Tandem gap = 15
        Flush output = 1000
        Q-value = 0.75
        Anchorpoints = 3
        Probability cutoff = 0.01
        Cloud filtering method = Binomial
        Level 2 only = true
        Use family = true
        Write statistics = false
        Alignment method = GreedyGraphbased4
        Multiple hypothesis correction = Bonferroni
        Number of threads = 4
        Compare aligners = false
        Combined search (first collinear, then cloud)
        Visualize GHM.png = false
        Visualize Alignment = false
        Verbose output = true
************ END i-AdDHoRe parameters *********

Creating dataset...                     done. (time: 0.523861s)
Mapping gene families...                done. (time: 0.253818s)
Remapping tandem duplicates...  done. (time: 0.0680161s)
Writing genelists file...               done. (time: 0.177472s)
Hybrid Search (first Collinear, then Cloud Search on what is remaining in GHM)
Level 2 multiplicon detection...bufSize= 223
getPackSize()= 0

anchorPoints.size= 11

sizeof(x_objectID)=4
sizeof(y_objectID)= 4
sizeof(begin_x)= 4
sizeof(end_x)= 4
sizeof(begin_y)= 4
sizeof(end_y)= 4
sizeof(random_probability)= 8

sizeof(size)= 4
sizeof(int)= 4

size of pack(buffer) statement = 17
i-adhore: /i-adhore-3.0.01/src/SynthenicCloud.cpp:320: int SynthenicCloud::pack(char*) const: Assertion `bufSize==getPackSize()' failed.
[vlogin2:15756] *** Process received signal ***
[vlogin2:15756] Signal: Aborted (6)
[vlogin2:15756] Signal code:  (-6)
[vlogin2:15756] [ 0] /lib64/libpthread.so.0(+0xf630)[0x2ab66d559630]
[vlogin2:15756] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2ab66e4d8387]
[vlogin2:15756] [ 2] /lib64/libc.so.6(abort+0x148)[0x2ab66e4d9a78]
[vlogin2:15756] [ 3] /lib64/libc.so.6(+0x2f1a6)[0x2ab66e4d11a6]
[vlogin2:15756] [ 4] /lib64/libc.so.6(+0x2f252)[0x2ab66e4d1252]
[vlogin2:15756] [ 5] /bin/i-adhore(_ZNK14SynthenicCloud4packEPc+0x696)[0x428a86]
[vlogin2:15756] [ 6] /bin/i-adhore(_ZN14SynthenicCloud19packSynthenicCloudsERKSt6vectorIPS_SaIS1_EEPc+0x38)[0x428e28]
[vlogin2:15756] [ 7] /bin/i-adhore(_ZN7DataSet23parallelLevel2ADHoReDynEv+0x278)[0x41fbc8]
[vlogin2:15756] [ 8] /bin/i-adhore(main+0x29e)[0x41a19e]
[vlogin2:15756] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ab66e4c4555]
[vlogin2:15756] [10] /bin/i-adhore[0x41a6cf]
[vlogin2:15756] *** End of error message ***
Aborted


And below I will add the complete output of my cmake command.
I wanted to add this specifically because I saw the same report from someone on the internet and I noticed that I don't get the Found MPI: TRUE line. Here are those lines from the report I found (which I can not be sure if clouds worked for them but still something looks better there) :

-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")

So I am not sure if cmake actually correctly finds my MPI or not.

And on this line, that report finds only the first libmpi_cxx.so, but mine also finds libmpi.so as well following a ;
-- Found MPI_CXX: /usr/lib64/openmpi3/lib/libmpi_cxx.so;/usr/lib64/openmpi3/lib/libmpi.so

And my Found MPI lines don't state the MPI version, while that other report always states the version.

Before starting the install, I only load the openmpi module's bin folder using this command
export PATH=$PATH:/usr/lib64/openmpi3/bin/
Then it doesn't give the MPI not found errors.
Normally we used MPI before on our cluster and it worked fine.
Here is my cmake output:


-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /apps/gcc/7.4.0/bin/cc
-- Check for working C compiler: /apps/gcc/7.4.0/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /apps/gcc/7.4.0/bin/c++
-- Check for working CXX compiler: /apps/gcc/7.4.0/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found MPI_C: /usr/lib64/openmpi3/lib/libmpi.so
-- Found MPI_CXX: /usr/lib64/openmpi3/lib/libmpi_cxx.so;/usr/lib64/openmpi3/lib/libmpi.so
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7")
-- Found PNG: /usr/lib64/libpng.so (found version "1.5.13")
-- Configuring done
-- Generating done
-- Build files have been written to: /proj/fatih/IADORE3/i-adhore-3.0.01/build

And following this, make and make install commands don't raise any warning or error messages, either.

Thanks!

@bedroesb
Copy link
Member

@FatihSarigol Thank you very much for this in depth information. I only was involved in making this public and generating the Docker image. I requested people that have experience with I-adhore to have a look at this issue.

Small remark on my side: if you use singularity you don't need root access to run the container.

@PSB-mibel
Copy link
Member

We can't really replicate the issue locally I'm afraid. However, I would like to note that this cloud-mode was only added as a 'last' feature to the software release of I-ADHoRe 3.0, and that nobody (internally where the software was developed) has used the cloud-mode due to the too generalized output it produces. So if possible, I would suggest limiting yourself to the alignment-mode of I-ADHoRE, which is the mode used by everyone (we know) who uses the software.

@FatihSarigol
Copy link
Author

@bedroesb Thank you both for your reply! I think I figured out how to use Singularity and pulled i-adhore from docker hub, but eventually failed to find a solution to this issue below, which is as far as I understand related to the way Singularity is installed to our HPC requiring root access. If you already know how to solve it, that would be great help:

FATAL: while extracting i-adhore_latest.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled

@PSB-mibel Thank you for your reply! We were mainly interested in i-adhore because of the cloud mode. But it should still be working fine if I could manage to install it without any issue, right?

Thanks both!!

@taprs
Copy link

taprs commented Jan 18, 2024

Hi all,

I am getting the same assertion error as here with local installation, in single-threaded mode as well. Not sure what data might be helpful for debugging, but I am happy to provide anything upon request.

Singularity approach worked well for me like that:

$ singularity pull docker://vibpsb/i-adhore
$ singularity shell -B /path/to/data i-adhore_latest.sif
Singularity> i-adhore file.conf 

Cheers,
Nikita

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants