Dockerfile fails to compile (poetry install, step 27)

[wariobrega]

Dear SCope Team,

First of all, congratulations for SCope and SCENIC such nice tools, I find them incredibly useful for my analyses.

I however have an issue when trying to rebuild SCope using the Dockerfile you provided on a local machine in my Institute

Before I gon any further, a couple of technical details:

Machine OS: Ubuntu Server 18.04
Docker version: 20.10.7

The issue occurs at line 48

RUN cd /app/opt && poetry install

it seems that all the packages that should be installed by the poetry tools fails to compile

I attach here the Docker-build log from step 27 (the moment in which Docker fails) onwards.
Docker-build.log

On a side note, I found that running each command inside docker manually works (i.e., I create an empty Debian Docker container and I copy paste all commands that yu wrote in theDockerfile)

I am scratching my head around the issue but I cannot find a way to solve it. Can you point me to the right direction?

Thanks,

Daniele

[KrisDavie]

Hey Daniele,

Unfortunately, due to a large rewrite effort, that fact that the Docker setup was contributed by an external collaborator and also not currently used by us, the existing Dockerfile has been somewhat left behind. We intend to fix this with the next "big" version of SCope, although I realize this doesn't help you right now.

I've tried to also do some debugging, but have come to the same conclusion as you, that for some reason, running the commands manually within the container seems to work, but the build is broken. I managed to overcome the llvm issues, but ran into problems with scipy refusing to install via the wheel.

Although not optimal, is running using a conda environment locally an option for you at the moment? This is how we currently run our main instance. Sorry I haven't been more help.

Cheers,

Kris

[wariobrega]

Dear Kris,

Thanks a lot for your answer.

I have not tried to do install SCope through Conda, do you have a .yaml file I can use? Also, do I need to install some dependency system wide? I was using Docker because I fear that nodejs installation may broke other tools and dependencies.

If you have some instruction to share on your local install that differs from the one listed on the README page, I'll be glad to test them on!

Cheers,

Daniele

[KrisDavie]

Hey Daniele,

I don't have a .yaml but poetry should lock the versions of the python packages you need. The benefit of conda here is that you can actually install both python and nodejs within the conda env which will prevent issues with system dependencies.

This conda create command should work for master: conda create -n scope "python=3.7.4" "nodejs=10.13.0", I've just checked and these are the versions we're using on our main deployment. Then just make sure to conda activate scope before you do the install instructions in the README.md.

If you have issues, just let me know and I'll see where I can help out.

Cheers,

Kris

[wariobrega]

Helo Kris,

I have tried following the instructions listed on the site in a conda environment created using your specifications and the npm install command fails.

I attach here the (quite long) error log, it seems to be a problem of some package that is corrupted (at the beginning of the log) that subsequently create the error cascade.

Can you look it up and point me to the right fix?

Thanks!

Cheers,

Daniele

scope_npm_error.log

[KrisDavie]

Hey Daniele,

I've seen this sometimes be cause by the package.lock files from npm, can you try removing these two files from the root ofthe repository rm package-lock.json opt/scopeserver/bindserver/package-lock.json and see if that works? You may also need to remove the existing node module files rm -r node_modules opt/scopeserver/bindserver/node_modules.

Cheers,

Kris

[wariobrega]

Hi kris,

After a few workarounds (machine-related issues), I managed to successfully complete the npm-install command.

However, npm run scope fails to launch, apparently it seems that the scope-server is not installed.

I honestly don't know where to look up, I even installed poetry as a dependency and re-run the installation on a separate machine.

I attach here both the npm-install log and the npm-run-scope error logs.

Thanks again,

Daniele
npm_run_scope.log
npm_install.log

[KrisDavie]

Hey Daniele,

I'm really sorry about all of these problems - this is of course not the experience we want people to have when trying to run SCope. It looks like due to various updates, certain parts of the install are not working. I have worked out a set of commands here, that from a fresh start will get SCope running for you. Would you mind giving these a go?

git clone https://github.com/aertslab/SCope
cd SCope

# Install miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -p ./miniconda3 
source miniconda3/bin/activate
conda create -n scope "python=3.7.4" "nodejs=10.13.0"
conda activate scope

# Fix issues with package versions
rm opt/scopeserver/bindserver/package-lock.json
sed -i 's/"bundled": true/"bundled": false/g' package-lock.json

# Install and build client files
npm install
npm run build

# Move client files to a unique directory - prevents leaking files when using python http.server
mkdir host
cd host
ln -s ../assets ./
ln -s ../images ./
ln -s ../index.html ./
ln -s ../index.js ./
ln -s ../src ./

# Use python to host the frontend
python -m http.server 55850

#In another terminal run the backend
cd SCope
source miniconda3/bin/activate
conda activate scope
cd opt
TMPDIR=/tmp DATABASE_URL="sqlite:///scope.sqlite" SCOPE_CONFIG=../config.json poetry run hypercorn --bind 0.0.0.0:55854 main:scope_api

Cheers,

Kris

[wariobrega]

Hi @KrisDavie,

No worries, I understand that these problems may occur, especially with nodejs, and they are quite pesky to resolve.

I followed your instructions carefully but, alas, the npm install still fails to build.

here is the error log:

(scope) caposquared@RainbowUbuntu:~/SCope$ npm install                                                                                                          
> [email protected] preinstall /home/caposquared/SCope            
> npm i shelljs command-exists && node bin/install.js                                                                                                              
npm ERR! code ETARGET                                       
npm ERR! notarget No matching version found for [email protected]       
npm ERR! notarget In most cases you or one of your dependencies are requesting               
npm ERR! notarget a package version that doesn't exist.
                                      
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/caposquared/.npm/_logs/2022-01-20T14_03_56_297Z-debug.log                                                              
npm ERR! code ELIFECYCLE
npm ERR! errno 1      
npm ERR! [email protected] preinstall: `npm i shelljs command-exists && node bin/install.js`
npm ERR! Exit status 1
npm ERR!                                                                              
npm ERR! Failed at the [email protected] preinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?            
                           
npm ERR! A complete log of this run can be found in:                                                       
npm ERR!     /home/caposquared/.npm/_logs/2022-01-20T14_03_56_384Z-debug.log

A few things that I tried:

1. wget on all the grb-bus instances from my machine seems to work as a user (thought it may be some incorrect URL, especially this one https://github.com/aertslab/grpc-bus/releases/download/v1.0.2/grpc-bus-1.0.2.tgz since it seems the only matching grpc-bus version
2. removing the package-lock.json and run npm install seems to work, but npm run build fails

I sadly have no confidence with npm so these were workarounds I tried as simple sanity checks, but if you have other suggestions I'll be glad to know them

I attach here the two error logs generated by the npm install command.
2022-01-20T14_03_56_384Z-debug.log
2022-01-20T14_03_56_297Z-debug.log

In case this process becomes too painful to debug, we can also meet and check it together, no problem for me.

Ciao,

Daniele

[wariobrega]

Ciao Kris,

Thanks a lot for your help, your time and for all the precious info that you gave me, now everything is a little bit clearer :)

I attach here a small .md file that sums up all the operations that we did yesterday. I hope that I did not forget something.

Feel free to edit and distribute it as much as you want, hopefully it can be useful for other SCope users as well!

Waiting for the new Scope to unroll, keep up with the good work!

install_scope.md

Ciao,

Daniele