Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TASK-6442 - Add Resources Management functionality #2535

Open
wants to merge 55 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
0ee51eb
core: add ResourceManager and JUnit tests, #TASK-7047, #TASK-6442
jtarraga Oct 3, 2024
c6f84ef
catalog: create default RESOURCE folder for studies, #TASK-6442
pfurio Oct 7, 2024
77cccad
core: remove old FileCreateParams, #TASK-6442
pfurio Oct 7, 2024
b754a47
core: add an MD5 check when the resource manager downloads a file, #T…
jtarraga Oct 10, 2024
7923785
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 10, 2024
6fb08b9
analysis: implement the tool DownloadResourcesTool to download all an…
jtarraga Oct 11, 2024
2fd7678
core: minor improvements
imedina Oct 11, 2024
0e80c4d
core: first download all resources to a temp folder, then copy them t…
jtarraga Oct 14, 2024
8673ebd
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 14, 2024
3baf7c9
core: rename DownloadResourceTool to ResourceFetcherTool, #TASK-7088,…
jtarraga Oct 14, 2024
8885f30
master: call the ResourceFetcherTool by the monitor service on initia…
jtarraga Oct 14, 2024
774c286
client: generate clients after renaming some fields, #TASK-7088, #TAS…
jtarraga Oct 14, 2024
2b6bfa0
core: use .md5 file to check if the resource file must be downloaded,…
jtarraga Oct 14, 2024
d6007c9
app: add migration to create missing RESOURCES folder, #TASK-6442
pfurio Oct 16, 2024
c4b872c
analysis: fetch all resources only is exececuted by admins, #TASK-708…
jtarraga Oct 17, 2024
6629f3f
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 17, 2024
90e7280
catalog: add resource checks, #TASK-6442
pfurio Oct 17, 2024
701a076
catalog: fix several tests, #TASK-6442
pfurio Oct 18, 2024
297009c
analysis: improve resource suppport and add resource exception, #TASK…
jtarraga Oct 18, 2024
fa2cba7
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 18, 2024
4fad723
analysis: simplify ResourceManager code, #TASK-7047, #TASK-6442
jtarraga Oct 22, 2024
0791fe0
catalog: fix some tests, #TASK-6442
pfurio Oct 22, 2024
e1b5b70
Merge branch 'TASK-6442' of github.com:opencb/opencga into TASK-6442
pfurio Oct 22, 2024
973d72d
analysis: update Exomiser analysis to use ResourceManager, #TASK-7125…
jtarraga Oct 23, 2024
379a0bf
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 23, 2024
9807f37
analysis: add the action UNZIP to unzip Exomiser files after download…
jtarraga Oct 23, 2024
97ce73f
analysis: update mutational signature analysis according to the new r…
jtarraga Oct 23, 2024
ff513c2
core: change resource metadata to be independent/standalone, and rena…
jtarraga Oct 24, 2024
990b757
analysis: remove ResourceUtils and update relatedness analysis by usi…
jtarraga Oct 24, 2024
159e76a
analysis: fix exomiser resource path, #TASK-7125, #TASK-6442
jtarraga Oct 24, 2024
481f2e8
analysis: fix mutational signature resource path, #TASK-7127, #TASK-6442
jtarraga Oct 24, 2024
4a76f4b
analysis: mount exomiser data resources as readonly, #TASK-7125, #TAS…
jtarraga Oct 24, 2024
ba9d0bc
core: add new RESOURCES constant, #TASK-6442
pfurio Oct 28, 2024
eb37a22
catalog: allow filtering by new resource field, #TASK-6442
pfurio Oct 30, 2024
9aafcb3
Merge branch 'develop' into TASK-6442
pfurio Oct 30, 2024
e1ed083
app: fix uri generation in migration code, #TASK-6442
pfurio Oct 31, 2024
f7e0b53
catalog: improve resource manager, #TASK-7047, #TASK-6442
jtarraga Nov 6, 2024
1184713
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Nov 6, 2024
86b0194
catalog: avoid moving non-resource file to Resource folder, #TASK-6442
pfurio Nov 28, 2024
62db7e8
Merge branch 'develop' into TASK-6442
pfurio Nov 28, 2024
9434fe1
test: update JUnit tests, #TASK-7047, #TASK-6442
jtarraga Dec 3, 2024
65e51c7
catalog: improve resource manager, #TASK-7088, #TASK-6442
jtarraga Dec 4, 2024
023f04d
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Dec 4, 2024
3d66280
core: implement ResourceConfiguration and update according to this, #…
jtarraga Dec 9, 2024
70ee362
catalog: use constants where possible, #TASK-6442
pfurio Dec 10, 2024
0eb2ad3
Merge branch 'develop' into TASK-6442
pfurio Dec 10, 2024
9d8ddb2
catalog: improve ResourceManager, #TASK-7047, #TASK-6442
jtarraga Dec 11, 2024
e0d0f9f
catalog: remove unused import, #TASK-7047, #TASK-6442
jtarraga Dec 11, 2024
eed1973
analysis: update relatedness/family-qc according to resource manager …
jtarraga Dec 11, 2024
dc0de9c
analysis: update exomiser according to resource manager changes, #TAS…
jtarraga Dec 11, 2024
6e6141c
analysis: update mutational signature according to resource manager c…
jtarraga Dec 11, 2024
44d3c3f
analysis: update fetch resource tool according to resource manager ch…
jtarraga Dec 11, 2024
8aed078
test: fix JUnit test and update configuration files, #TASK-7047, #TAS…
jtarraga Dec 11, 2024
943e4de
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Dec 11, 2024
a5cee2a
client: generate clients, #TASK-6442
jtarraga Dec 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
package org.opencb.opencga.analysis;

import org.apache.commons.collections4.CollectionUtils;
import org.opencb.commons.datastore.core.Query;
import org.opencb.commons.datastore.core.QueryOptions;
import org.opencb.opencga.catalog.db.api.FileDBAdaptor;
import org.opencb.opencga.catalog.db.api.ProjectDBAdaptor;
import org.opencb.opencga.catalog.exceptions.CatalogException;
import org.opencb.opencga.catalog.managers.CatalogManager;
import org.opencb.opencga.catalog.managers.FileManager;
import org.opencb.opencga.catalog.managers.JobManager;
import org.opencb.opencga.catalog.utils.CatalogFqn;
import org.opencb.opencga.core.exceptions.ToolException;
import org.opencb.opencga.core.models.JwtPayload;
import org.opencb.opencga.core.models.common.Enums;
import org.opencb.opencga.core.models.file.File;
import org.opencb.opencga.core.models.job.Job;
import org.opencb.opencga.core.models.project.Project;
import org.opencb.opencga.core.response.OpenCGAResult;

import java.io.BufferedReader;
Expand Down Expand Up @@ -193,4 +199,22 @@ public static String getJobFileRelativePath(String path) {
}
return path.substring(index + 5);
}

public static String getAssembly(CatalogManager catalogManager, String studyId, String sessionId) throws CatalogException {
String assembly = "";
OpenCGAResult<Project> projectQueryResult;

JwtPayload jwtPayload = catalogManager.getUserManager().validateToken(sessionId);
CatalogFqn studyFqn = CatalogFqn.extractFqnFromStudy(studyId, jwtPayload);
String organizationId = studyFqn.getOrganizationId();

projectQueryResult = catalogManager.getProjectManager().search(organizationId, new Query(ProjectDBAdaptor.QueryParams.STUDY.key(), studyId),
new QueryOptions(QueryOptions.INCLUDE, ProjectDBAdaptor.QueryParams.ORGANISM.key()), sessionId);
if (CollectionUtils.isNotEmpty(projectQueryResult.getResults())
&& projectQueryResult.first().getOrganism() != null
&& projectQueryResult.first().getOrganism().getAssembly() != null) {
assembly = projectQueryResult.first().getOrganism().getAssembly();
}
return assembly;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -121,4 +121,27 @@ public static String getToolDefaultVersion(String toolId, Configuration configur
}
return defaultVersion;
}


public static AnalysisTool getAnalysisTool(String toolId, String version, Configuration configuration) throws ToolException {
for (AnalysisTool tool : configuration.getAnalysis().getTools()) {
if (toolId.equals(tool.getId())) {
if (StringUtils.isEmpty(version) || version.equals(tool.getVersion())) {
return tool;
}
}
}
throw new ToolException("Missing analysis tool (ID = " + toolId + ", version = " + version + ") in configuration file");
}

public static String getToolResource(String toolId, String version, String resourceKey, Configuration configuration)
throws ToolException {
// Get resources from the configuration file
AnalysisTool tool = ConfigurationUtils.getAnalysisTool(toolId, version, configuration);
if (!tool.getResources().containsKey(resourceKey)) {
throw new ToolException("Error getting resource " + resourceKey + " of analysis tool (ID = " + toolId + ", version = "
+ version + "): it does not exist in the configuration file");
}
return tool.getResources().get(resourceKey);
}
}

This file was deleted.

Loading
Loading