Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

diva_to_gatingset: option required to *not* use compensation matrix from xml #128

Open
RPdavies opened this issue Mar 23, 2021 · 13 comments

Comments

@RPdavies
Copy link

Hi,

In cases where a compensation matrix was not created in Diva, diva_to_gatingset attempts to find one in the xml file and returns an error: Error in solve.default(comp) : 'a' (2 x 4) must be square.

This makes importing xml gates from Diva impossible in many situations. Let me know if you require an example xlm / fcs set.

@mikejiang
Copy link
Member

yeah, send the link or data to [email protected]. Thanks!

@RPdavies
Copy link
Author

No problem, I've sent it over just now. Please let me know if the link didn't work for some reason. Thank you!

@mikejiang
Copy link
Member

There is currently no elegant way to decide whether xml compensation should/can be used since compensation for this xml is actually present, but as dummy setting, thus caused the parse to be confused.

<compensation>
<compensation_coefficient>0.0</compensation_coefficient>
<compensation_coefficient>1.0</compensation_coefficient>
</compensation>

I've pushed the easy workaround as you suggested, i.e. optionally disable it

  gs <- diva_to_gatingset(ws, name = 1, xml_compensation_enabled = FALSE)

@RPdavies
Copy link
Author

RPdavies commented Apr 1, 2021

Awesome, that does the job. Look forward to using CytoML!

@nvannest-absci
Copy link

nvannest-absci commented May 24, 2021

I also have a dummy compensation issue in my xml file.
Screen Shot 2021-05-26 at 11 40 16 AM

I tried setting xml_compensation_enabled to either TRUE or FALSE and both times I received the same error. Are there alternative solutions to this problem?

Screen Shot 2021-05-24 at 2 58 30 PM

I am wondering if it might be easier to create a function that removes dummy compensation data from diva xml files. I tried removing the code listed above from my xml file but I received the following error.
Screen Shot 2021-05-26 at 11 42 48 AM

@mikejiang
Copy link
Member

Can you show the error and traceback() output when you set xml_compensation_enabled = F ?

@nvannest-absci
Copy link

nvannest-absci commented May 27, 2021

Screen Shot 2021-05-27 at 2 19 23 PM

I don't know much about these functions, but it seems like diva_to_gatingset is not passing xml_compensation_enabled = F to .parseDivaWorkspace when it is called. I could be misinterpreting things however.

@mikejiang
Copy link
Member

You may need to pull the latest version from github.

@nvannest-absci
Copy link

nvannest-absci commented Jun 1, 2021

I am currently having trouble running the diva_to_gatingset function from the CytoML package

I am using CytoML version 2.4.0. available through biocManager
My experiment lacks any compensation, but my XML from Diva includes placeholder values. An example is shown below:

> library(flowWorkspace)
> library(CytoML)
> setwd("/Users/user1/Desktop/DummyProject_Folder/DummyExp")
> flowData <- getwd()

> gatingFile <- list.files(flowData, pattern=".xml",full = TRUE)
> ws <- open_diva_xml(gatingFile)
> ws
> diva_get_sample_groups(ws)
> gs <- diva_to_gatingset(ws)
> sampleNames(gs)
> gs_get_pop_paths(gs)
> plotGate(gs[[1]])

> sessionInfo()

Diva Workspace Version  Version 9.1.2 
File location:  /Users/user1/Desktop/DummyProject_Folder/DummyExp
File name:  DummyExp.xml 
Workspace is open. 

Groups in Workspace
> diva_get_sample_groups(ws)
> gs <- diva_to_gatingset(ws)
Choose which group of samples to import:


1: DummySpecimen1

> Selection: 1
Parsing 2 samples
loading data: /Users/nvannest/Desktop/ACE openCyto Test/DummyProject_Folder/DummyExp_Exp_Export/DummyExp_DummySpecimen1_DummyTube1.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
Error in solve.default(comp) : 'a' (7 x 14) must be square )

When I try to install the version of CytoML available through github I get an installation error

> devtools::install_github("RGLab/CytoML")
Downloading GitHub repo RGLab/CytoML@HEAD
These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All                                  
2: CRAN packages only                   
3: None                                 
4: robustbase  (0.93-7 -> 0.93-8) [CRAN]
5: matrixStats (0.58.0 -> 0.59.0) [CRAN]
6: ks          (1.13.0 -> 1.13.1) [CRAN]

> Enter one or more numbers, or an empty line to skip updates: 1
robustbase  (0.93-7 -> 0.93-8) [CRAN]
matrixStats (0.58.0 -> 0.59.0) [CRAN]
ks          (1.13.0 -> 1.13.1) [CRAN]
Installing 3 packages: robustbase, matrixStats, ks

  There are binary versions available but the source versions are later:
            binary source needs_compilation
robustbase  0.93-7 0.93-8              TRUE
matrixStats 0.58.0 0.59.0              TRUE
ks          1.13.0 1.13.1              TRUE

> Do you want to install from sources the packages which need compilation? (Yes/no/cancel) no
trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/robustbase_0.93-7.tgz'
Content type 'application/x-gzip' length 3123297 bytes (3.0 MB)
==================================================
downloaded 3.0 MB

trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/matrixStats_0.58.0.tgz'
Content type 'application/x-gzip' length 2019790 bytes (1.9 MB)
==================================================
downloaded 1.9 MB

trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/ks_1.13.0.tgz'
Content type 'application/x-gzip' length 1851632 bytes (1.8 MB)
==================================================
downloaded 1.8 MB


The downloaded binary packages are in
	/var/folders/_g/xp8hp7mn02z122v2092lzw499zps22/T//RtmpcPvBBy/downloaded_packageschecking for file/private/var/folders/_g/xp8hp7mn02z122v2092lzw499zps22/T/RtmpcPvBBy/remotes132e432e8405d/RGLab-CytoML-d544a00/DESCRIPTION...preparingCytoML:checking DESCRIPTION meta-information ...cleaning srcchecking for LF line-endings in source and make files and shell scriptschecking for empty or unneeded directories
   OmittedLazyDatafrom DESCRIPTION
     NB: this package now depends on R (>= 3.5.0)
     WARNING: Added dependency on R >= 3.5.0 because serialized objects in
     serialize/load version 3 cannot be read in older versions of R.
     File(s) containing such objects:CytoML/tests/testthat/expect_result/flowJoWorkspace_expect.rds’
─  buildingCytoML_2.5.1.tar.gzWarning in sprintf(gettext(fmt, domain = domain), ...) :
     one argument not used by format 'invalid uid value replaced by that for user 'nobody''
   Warning: invalid uid value replaced by that for user 'nobody'
   Warning in sprintf(gettext(fmt, domain = domain), ...) :
     one argument not used by format 'invalid gid value replaced by that for user 'nobody''
   Warning: invalid gid value replaced by that for user 'nobody'
   
* installing *source* packageCytoML...
** using staged installation
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether clang++ -mmacosx-version-min=10.13 -std=gnu++11 accepts -g... yes
checking for gcc... clang -mmacosx-version-min=10.13
checking whether we are using the GNU C compiler... yes
checking whether clang -mmacosx-version-min=10.13 accepts -g... yes
checking for clang -mmacosx-version-min=10.13 option to accept ISO C89... none needed
configure: setting xml2 flags...
configure: No directory was specified for --with-xml2. Trying to find xml2 using other methods.
checking for xml2-config... /usr/bin/xml2-config
configure: setting cytolib-ml commandline tool path...
configure: Using the following compilation and linking flags
configure:    PKG_CPPFLAGS=-I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -ftemplate-depth=900
configure:    PKG_LIBS=-L/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib -lxml2 -lz -lpthread -licucore -lm 
configure:    CYTOLIBML_BIN=/usr/local/bin
configure: creating ./config.status
config.status: creating src/Makevars
config.status: creating R/cytolibml_bin_path.R
** libs
clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -DROUT -I../inst/include/ -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -ftemplate-depth=900 -DBOOST_NO_AUTO_PTR  -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/RProtoBufLib/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rhdf5lib/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/RcppArmadillo/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/RcppParallel/include' -I'/Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include' -I/usr/local/include   -fPIC  -Wall -g -O2  -c RcppExports.cpp -o RcppExports.o
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/CytoML.h:4:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include/flowWorkspace.h:5:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include/cytolib/GatingSet.hpp:14:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include/cytolib/GatingHierarchy.hpp:17:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/cytolib/include/cytolib/populationTree.hpp:14:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/graph/adjacency_list.hpp:255:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/graph/detail/adjacency_list.hpp:37:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/graph/adjacency_iterator.hpp:13:
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/detail/iterator.hpp:13:1: warning: This header is deprecated. Use <iterator> instead. [-W#pragma-messages]
BOOST_HEADER_DEPRECATED("<iterator>")
^
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/config/header_deprecated.hpp:23:37: note: expanded from macro 'BOOST_HEADER_DEPRECATED'
# define BOOST_HEADER_DEPRECATED(a) BOOST_PRAGMA_MESSAGE("This header is deprecated. Use " a " instead.")
                                    ^
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/BH/include/boost/config/pragma_message.hpp:24:34: note: expanded from macro 'BOOST_PRAGMA_MESSAGE'
# define BOOST_PRAGMA_MESSAGE(x) _Pragma(BOOST_STRINGIZE(message(x)))
                                 ^
<scratch space>:12:2: note: expanded from here
 message("This header is deprecated. Use " "<iterator>" " instead.")
 ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/CytoML.h:4:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include/flowWorkspace.h:6:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/flowWorkspace/include/flowWorkspace/pairVectorRcppWrap.h:15:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/RcppArmadillo/include/RcppArmadillo.h:34:
In file included from /Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rcpp/include/Rcpp.h:57:
/Library/Frameworks/R.framework/Versions/4.1/Resources/library/Rcpp/include/Rcpp/DataFrame.h:136:18: warning: unused variable 'data' [-Wunused-variable]
            SEXP data = Parent::get__();
                 ^
In file included from RcppExports.cpp:4:
In file included from ./../inst/include/CytoML.h:5:
In file included from ../inst/include/CytoML/openWorkspace.hpp:12:
../inst/include/CytoML/winFlowJoWorkspace.hpp:435:30: warning: '/*' within block comment [-Wcomment]
           * "*[local-name()='edge']/*[local-name()='vertex']" is for ellipsoidGate
                                    ^
../inst/include/CytoML/winFlowJoWorkspace.hpp:866:18: error: no member named 'setDataOnlyFlag' in 'cytolib::scaleTrans'
                                    curTran->setDataOnlyFlag(true);
                                    ~~~~~~~  ^
../inst/include/CytoML/winFlowJoWorkspace.hpp:882:16: error: no member named 'setDataOnlyFlag' in 'cytolib::scaleTrans'
                                  curTran->setDataOnlyFlag(false);
                                  ~~~~~~~  ^
3 warnings and 2 errors generated.
make: *** [RcppExports.o] Error 1
ERROR: compilation failed for packageCytoML* removing/Library/Frameworks/R.framework/Versions/4.1/Resources/library/CytoML* restoring previous/Library/Frameworks/R.framework/Versions/4.1/Resources/library/CytoMLWarning message:
In i.p(...) :
  installation of package/var/folders/_g/xp8hp7mn02z122v2092lzw499zps22/T//RtmpcPvBBy/file132e439dffd5e/CytoML_2.5.1.tar.gzhad non-zero exit status

Here is additional information about my setup:

R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

BiocManager version 1.30.15

> tools:::.BioC_version_associated_with_R_version()
[1] ‘3.13

Any help would be very appreciated. Thanks!

@mikejiang
Copy link
Member

try to update cytolib and flowWorkspace from github first

@nvannest-absci
Copy link

I've updated cytolib and flowWorkspace directly from github and continue to get the same error

Input:

library(flowWorkspace)
library(CytoML)

setwd("/Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export")
flowData <- getwd()

gatingFile <- list.files(flowData, pattern=".xml",full = TRUE)
ws <- open_diva_xml(gatingFile)
ws
diva_get_sample_groups(ws)
gs <- diva_to_gatingset(ws)
sampleNames(gs)
gs_get_pop_paths(gs)
plotGate(gs[[1]])

Output:

> setwd("/Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export")
The working directory was changed to /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export inside a notebook chunk. The working directory will be reset when the chunk is finished running. Use the knitr root.dir option in the setup chunk to change the working directory for notebook chunks.> flowData <- getwd()
> 
> gatingFile <- list.files(flowData, pattern=".xml",full = TRUE)
> ws <- open_diva_xml(gatingFile)
> ws
Diva Workspace Version  Version 9.1.2 
File location:  /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export 
File name:  DummyExp.xml 
Workspace is open. 

Groups in Workspace
> diva_get_sample_groups(ws)
> gs <- diva_to_gatingset(ws)
Choose which group of samples to import:


1: DummySpecimen1

Selection: 1
Parsing 2 samples
loading data: /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export/DummyExp_DummySpecimen1_DummyTube1.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
Error in solve.default(comp) : 'a' (7 x 14) must be square

@mikejiang
Copy link
Member

how about set xml_compensation_enabled = F?

@nvannest-absci
Copy link

Success!!

Wow what a relief. Thanks so much for your help!

Selection: 1
Parsing 2 samples
loading data: /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export/DummyExp_DummySpecimen1_DummyTube1.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
computing data range
transforming ...
parsing gates ...
loading data: /Users/user1/Desktop/Flow_Data_Import_Test/DummyProject_Folder/DummyExp_Diva_Export/DummyExp_Diva_Exp_Export/DummyExp_DummySpecimen1_DummyTube2.fcs
swap cols: FSC-H:FSC-W
swap cols: SSC-H:SSC-W
Compensating
computing data range
transforming ...
parsing gates ...
done!
> sampleNames(gs)
[1] "DummyExp_DummySpecimen1_DummyTube1.fcs"
[2] "DummyExp_DummySpecimen1_DummyTube2.fcs"
> gs_get_pop_paths(gs)
[1] "root"         "/P1"          "/P1/P2"       "/P1/P2/P3"    "/P1/P2/P3/P4"
[6] "/P1/P2/P3/P5" "/P1/P2/P3/P6" "/P1/P2/P3/P8"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants