Viash 0.9.0

Viash 0.9.0 (2024-09-03): Restructure platforms into runners and engines

This release restructures the introduces changes to the Viash config:

• The platforms field is split into runners and engines
• The .functionality layer has been removed from the config and all fields have been moved to the top layer

Changes are made to sanitize the built config output and include additional relevant meta data.
The default multiple_sep has been changed from : to ; to avoid conflicts with paths like s3://foo/bar.

Implemented a proper way of caching dependency repositories. The cache is stored under ~/.viash/repositories.

BREAKING CHANGES

• runners and engines: The usage of platforms is deprecated and instead these are split into runners and engines (PR #510).
  The platforms field is still supported but will be removed in a future release.
  In brief, the native platform became a native engine and docker platform became a docker engine.
  Additionally, the native platform and docker platform became a executable runner, nextflow platform became a nextflow runner.
  The fields of docker platform is split between docker engine and docker runner: port, workdir, setup_strategy, and run_args (set to docker_run_args) are captured by the runner as they define how the component is run. The other fields are captured by the engine as they define the environment in which the component is run. One exception is chown which is rarely set to false and is now always enabled.

• arguments: Merge arguments into argument_groups during a json decode prepare step (PR #574). The --parse_argument_groups option from ns list and config view is deprecated as it is now always enabled.

• arguments: Change default multiple_sep from : to ; to avoid conflicts with paths like s3://foo/bar (PR #645).
  The previous behaviour of using multiple_sep: ":" can be achieved by adding a config mod to the _viash.yaml:

  config_mods: |
    .functionality.argument_groups[true].arguments[.multiple == true].multiple_sep := ":"

• functionality: Remove the functionality layer from the config and move all fields to the top layer (PR #649).

• computational requirements: Use 1000-base units instead of 1024-base units for memory (PR #686). Additionally, the memory units kib, mib, gib, tib, and pib are added to support 1024-base definitions.

• NextflowEngine: Swap the order of execution of runIf and filter when calling .run(). This means that runIf is now executed before filter (PR #660).

NEW FUNCTIONALITY

• export json_schema: Add a --strict option to output a subset of the schema representing the internal structure of the Viash config (PR #564).

• config view and ns list: Do not output internal functionality fields (#564). Additionally, add a validation that no internal fields are present when reading a Viash config file.

• project config: Add fields in the project config to specify default values for component config fields (PR #612). This allows for a more DRY approach to defining the same values for multiple components.

• dependencies: GitHub and ViashHub repositories now get properly cached (PR #699).
  The cache is stored in the ~/.viash/repositories directory using sparse-checkout to only fetch the necessary files.
  During a build, the cache is checked for the repository and if it is found and still up-to-date, the repository is not cloned again and instead the cache is copied to a temporary folder where the files are checked out from the sparse-checkout.

• ExecutableRunner: Add a ---docker_image_id flag to view the Docker image ID of a built executable (PR #741).

• viash ns query: Add a query filter that allows selecting a single component by its path in a namespace environment (PR #744).

• config schema: Add label & summary fields for Config, PackageConfig, argument groups, and all argument types (PR #743).

• NextflowEngine: Added runIf functionality to runEach (PR #660).

MINOR CHANGES

• testbenches: Add testbenches for local dependencies (PR #565).

• testbenches: Refactor testbenches helper functions to uniformize them (PR #565).

• logging: Preserve log order of StdOut and StdErr messages during reading configs in namespaces (PR #571).

• Java 21 support: Update Scala to 2.13.12 and update dependencies (PR #602).

• project config: Output the project config under the default name ProjectConfig instead of Project during schema export (PR #631). This is now important as the project config is now part of the component config. Previously this was overridden as the class name was ViashProject which was less descriptive.

• package config: Renamed project config to package config (PR #636). Now that we start using the config more, we came to the conclusion that "package" was better suited than "project".

• ns exec: Added an extra field {name} to replace {functionality-name} (PR #649). No immediate removal of the old field is planned, but it is deprecated.

• BashWrapper: Added meta-data field meta_name as a replacement for meta_functionality_name (PR #649). No immediate removal of the old field is planned, but it is deprecated.

• error message: Improve the error message when using an invalid field in the config (#PR #662). The error message now includes the field names that are not valid if that happens to be the case or otherwise a more general error message.

• config mods: Improve the displayed error message when a config mod could not be applied because of an invalid path (PR #672).

• docker_engine: Deprecate registry, organization and tag fields in the docker_engine (PR #712). Currently these are hardly ever used and instead the image field is used to specify the full image name.

• docker_engine: Add target_package field to the docker_engine (PR #712). This field, together with the target_organization is used to specify the full built container image name. The fields use proper fallback for the values set in the component config and package config.

• organization: Remove the organization field from the component config (PR #712). The value is now directly used by the docker_engine as a fallback from the target_organization field.

• ExecutableRunner: Add parameter docker_automount_prefix to allow for a custom prefix for automounted folders (PR #739).

• ExecutableRunner: Make Docker runtime arguments configurable via the ---docker_run_args argument (PR #740).

• export json_schema: Add arguments field to the Config schema (PR #755). Only for the non-strict version, the strict version of the viash config has these values merged into argument_groups.

• scala: Update Scala to 2.13.14 (PR #764).

• NextflowEngine: Also parse ${id} and ${key} aside from $id and $key as identifier placeholders for filenames (PR #756).

BUG FIXES

• __merge__: Handle invalid yaml during merging (PR #570). There was not enough error handling during this operation. Switched to the more advanced Convert.textToJson helper method.

• config: Anonymize paths in the config when outputting the config (PR #625).

• schema: Don't require undocumented fields to set default values and add the links and reference fields to functionality as they were not meant only to be in the project config (PR #636).

• export json_schema: Fix minor inconsistencies and make the strict schema stricter by adapting to what Viash will effectively return (PR #666).

• deprecation & removal warning: Improve the displayed warning where a deprecated or removed field could display a double '.' when it field was located at the root level (PR #671).

• resource path: Don't finalize the path field of a resource until it's written as part of building a component (PR #668).

• requirements: Improve the error message when a Python or R requirement uses a single quote in the .script field (PR #675).

• viash test: Fix Docker id between build and test components not being consistent when using a custom Docker registry (PR #679).

• MainNSBuildNativeSuite: Capture the error message when reading the configs so we can capture the expected warning message (PR #688).
  While almost all tests were already cleanly capturing their expected warning/error messages, this one was still remaining, resulting in warnings being shown in the output.

• runners & engines: When applying a filter on empty runners or engines, the fallback default native engine and executable runner respectively are set before applying the filter (PR #691).

• dependencies: Fix resolving of dependencies of dependencies (PR #701). The stricter build config was now lacking the necessary information to resolve dependencies of dependencies.
  We added it back as .build_info.dependencies in a more structured, anonymized way.

• dependencies: Fix the name field of repositories possibly being outputted in the build config (PR #703).

• symlinks: Allow following of symlinks when finding configs (PR #704). This improves symlink functionality for viash ns ... and dependency resolving.

• build_info: Correctly set the .build_info.executable to main.nf when building a component with a Nextflow runner (PR #720).

• vsh organization: ViashHub repositories now use vsh as the default organization (PR #718).
  Instead of having to specify repo: vsh/repo_name, you can now just specify repo: repo_name, which is now also the prefered way.

• testbenches: Add a testbench to verify dependencies in dependencies from scratch (PR #721).
  The components are built from scratch and the dependencies are resolved from the local repositories.

• docker_engine: Fix a bug in how the namespace separator is han...

Viash 0.9.0-RC7

Viash 0.9.0-RC7 (2024-08-13): General bugfixes and improvements

These are bug fixes and other improvements that solve some edge case issues and improve the overall user experience and usability of Viash.

BREAKING CHANGES

• NextflowPlatform: Swap the order of execution of runIf and filter when calling .run(). This means that runIf is now executed before filter (PR #660).

NEW FUNCTIONALITY

• ExecutableRunner: Add a ---docker_image_id flag to view the Docker image ID of a built executable (PR #741).

• viash ns query: Add a query filter that allows selecting a single component by its path in a namespace environment (PR #744).

• config schema: Add label & summary fields for Config, PackageConfig, argument groups, and all argument types (PR #743).

• NextflowPlatform: Added runIf functionality to runEach (PR #660).

MINOR CHANGES

• ExecutableRunner: Add parameter docker_automount_prefix to allow for a custom prefix for automounted folders (PR #739).

• ExecutableRunner: Make Docker runtime arguments configurable via the ---docker_run_args argument (PR #740).

• export json_schema: Add arguments field to the Config schema (PR #755). Only for the non-strict version, the strict version of the viash config has these values merged into argument_groups.

BUG FIXES

• platforms: Re-introduce the --platform and --apply_platform arguments to improve backwards compatibility (PR #725).
  When the argument is used, a deprecation warning message is printed on stderr.
  Cannot be used together with --engine or --runner and/or --apply_engine or --apply_runner.

• nextflow_runner: Fix refactoring error in the findStates() helper function (PR #733).

• viash ns exec: Fix "relative fields" outputting absolute paths (PR# 737). Additionally, improve path resolution when using the --src argument.

• viash ns: Fix viash tripping over its toes when it encounters multiple failed configs (PR #761). A dummy config was used as a placeholder, but it always used the name failed, so duplicate config names were generated, which we check for nowadays.

• bashwrapper: Fix an issue where running viash test which builds the test docker container would ignore test failures but subsequential runs would work correctly (PR #754).

Viash 0.9.0-RC6

Viash 0.9.0-RC6 (2024-06-17): Hotfix for docker image name generation

Fix an issue where docker image names were not generated correctly.

BUG FIXES

• docker_engine: Fix a bug in how the namespace separator is handled (PR #722).

Viash 0.9.0-RC5

Viash 0.9.0-RC5 (2024-06-13): Improvements for CI

Dependencies now use vsh as the default organization level. This means that the organization level is now optional in the repo field of the dependencies.
Improved how the docker image name is generated to be more predictable.

MINOR CHANGES

• resources_test: This field is removed again from the _viash.yaml as it was decided to impliment this temporary functionality using the info field (PR #711).

• docker_engine: Deprecate registry, organization and tag fields in the docker_engine (PR #712). Currently these are hardly ever used and instead the image field is used to specify the full image name.

• docker_engine: Add target_package field to the docker_engine (PR #712). This field, together with the target_organization is used to specify the full built container image name. The fields use proper fallback for the values set in the component config and package config.

• organization: Remove the organization field from the component config (PR #712). The value is now directly used by the docker_engine as a fallback from the target_organization field.

BUG FIXES

• build_info: Correctly set the .build_info.executable to main.nf when building a component with a Nextflow runner (PR #720).

• vsh organization: ViashHub repositories now use vsh as the default organization (PR #718).
  Instead of having to specify repo: vsh/repo_name, you can now just specify repo: repo_name, which is now also the prefered way.

• testbenches: Add a testbench to verify dependencies in dependencies from scratch (PR #721).
  The components are built from scratch and the dependencies are resolved from the local repositories.

Viash 0.9.0-RC4

Viash 0.9.0-RC4 (2024-05-29): Improvements for CI

These are mainly improvements for issues highlighted by running Viash in a CI environment.
Additionally, implemented a proper way of caching dependency repositories. The cache is stored under ~/.viash/repositories.

NEW FUNCTIONALITY

• dependencies: GitHub and ViashHub repositories now get properly cached (PR #699).
  The cache is stored in the ~/.viash/repositories directory using sparse-checkout to only fetch the necessary files.
  During a build, the cache is checked for the repository and if it is found and still up-to-date, the repository is not cloned again and instead the cache is copied to a temporary folder where the files are checked out from the sparse-checkout.

• resources_test: Add a resources_test field to the _viash.yaml to specify resources that are needed during testing (PR #709).
  Currently it is up to the user or CI to make sure these resources are available in the resources_test directory during testing.

BUG FIXES

dependencies: Fix resolving of dependencies of dependencies (PR #701). The stricter build config was now lacking the necessary information to resolve dependencies of dependencies.
We added it back as .build_info.dependencies in a more structured, anonymized way.

dependencies: Fix the name field of repositories possibly being outputted in the build config (PR #703).

symlinks: Allow following of symlinks when finding configs (PR #704). This improves symlink functionality for viash ns ... and dependency resolving.

Viash 0.9.0-RC3

Viash 0.9.0-RC3 (2024-04-26): Various bug fixes and minor improvements

Mainly fixes for code changes from previous release candidates. Some additional minor fixes and QoL improvements are included.

BREAKING CHANGES

• computational requirements: Use 1000-base units instead of 1024-base units for memory (PR #686). Additionally, the memory units kib, mib, gib, tib, and pib are added to support 1024-base definitions.

MINOR CHANGES

• error message: Improve the error message when using an invalid field in the config (#PR #662). The error message now includes the field names that are not valid if that happens to be the case or otherwise a more general error message.

• config mods: Improve the displayed error message when a config mod could not be applied because of an invalid path (PR #672).

BUG FIXES

• export json_schema: Fix minor inconsistencies and make the strict schema stricter by adapting to what Viash will effectively return (PR #666).

• deprecation & removal warning: Improve the displayed warning where a deprecated or removed field could display a double '.' when it field was located at the root level (PR #671).

• resource path: Don't finalize the path field of a resource until it's written as part of building a component (PR #668).

• requirements: Improve the error message when a Python or R requirement uses a single quote in the .script field (PR #675).

• viash test: Fix Docker id between build and test components not being consistent when using a custom Docker registry (PR #679).

• MainNSBuildNativeSuite: Capture the error message when reading the configs so we can capture the expected warning message (PR #688).
  While almost all tests were already cleanly capturing their expected warning/error messages, this one was still remaining, resulting in warnings being shown in the output.

• runners & engines: When applying a filter on empty runners or engines, the fallback default native engine and executable runner respectively are set before applying the filter (PR #691).

Viash 0.8.6

Viash 0.8.6 (2024-04-26): Bug fixes and improvements for CI

Fix some issues in some edge cases.
Add options for testing in a CI environment. Given that these options are not meant for general use, they are hidden from the help message.
Some improvements are made to run in Nextflow Fusion.

DOCUMENTATION

• docker setup strategy: Fix inconsistencies in the documentation (PR #657).

• repositories: Fix uri -> repo in the repositories documentation (PR #682).

NEW FUNCTIONALITY

• viash test and viash ns test: Add a hidden --dry_run option to build the tests without executing them (PR #676).

• viash test and viash ns test: Add a hidden --deterministic_working directory argument to use a fixed directory path (PR #683).

• component names: Verify that component namespace and name combinations are unique (PR #685).

BUG FIXES

• NextflowPlatform: Fix publishing state for output arguments with multiple: true (#638, PR #639).

• Executable: Check whether a multiple output file argument contains a wildcard (PR #639).

• NextflowPlatform: Fix a possible cause of concurrency issues (PR #669).

• Resources: Fix an issue where if the first resource is not a script, the resource is silently dropped (PR #670).

• Docker automount: Prevent adding a trailing slash to an automounted folder (PR #673).

• NextflowPlatform: Change the at-runtime generated nextflow process from an in-memory to an on-disk temporary file, which should cause less issues with Nextflow Fusion (PR #681).

Viash 0.9.0-RC2

Viash 0.9.0-RC2 (2024-02-23): Restructure the config and change some default values

The .functionality layer has been removed from the config and all fields have been moved to the top layer.
The default multiple_sep has been changed from : to ; to avoid conflicts with paths like s3://foo/bar.

BREAKING CHANGES

• arguments: Change default multiple_sep from : to ; to avoid conflicts with paths like s3://foo/bar (PR #645).
  The previous behaviour of using multiple_sep: ":" can be achieved by adding a config mod to the _viash.yaml:

  config_mods: |
    .functionality.argument_groups[true].arguments[.multiple == true].multiple_sep := ":"

• functionality: Remove the functionality layer from the config and move all fields to the top layer (PR #649).

MINOR CHANGES

• package config: Renamed project config to package config (PR #636). Now that we start using the config more, we came to the conclusion that "package" was better suited that "project".

• ns exec: Added an extra field {name} to replace {functionality-name} (PR #649). No immediate removal of the old field is planned, but it is deprecated.

• BashWrapper: Added meta-data field meta_name as a replacement for meta_functionality_name (PR #649). No immediate removal of the old field is planned, but it is deprecated.

BUG FIXES

• schema: Don't require undocumented fields to set default values and add the links and reference fields to functionality as they were not meant only to be in the project config (PR #636).

Viash 0.8.5

Viash 0.8.5 (2024-02-21): Bug fixes and documentation improvements

Fix a bug when building a test docker container which requires a test resource. Additional improvements for the website documentation and support for the latest version of Nextflow are added.

BUG FIXES

• nextflow runner: Fix an issue with current nextflow-latest (24.01.0-edge) where our supporting library passes a GString instead of a String and results in a type mismatch (PR #640).

• test resources: Make non-script test resources available during building of a docker container for viash test (PR #652).

DOCUMENTATION

• repositories: Improve the repository documentation with links and a overview table with links (PR #643).

Viash 0.9.0-RC1

Viash 0.9.0-RC1 (2024-01-26): Restructure platforms into runners and engines

This release restructures the platforms field into runners and engines.
Additionally changes are made to sanitize the built config output and include additional relevant meta data.

BREAKING CHANGES

• runners and engines: The usage of platforms is deprecated and instead these are split into runners and engines (PR #510).
  The platforms field is still supported but will be removed in a future release.
  In brief, the native platform became a native engine and docker platform became a docker engine.
  Additionally, the native platform and docker platform became a executable runner, nextflow platform became a nextflow runner.
  The fields of docker platform is split between docker engine and docker runner: port, workdir, setup_strategy, and run_args are captured by the runner as they define how the component is run. The other fields are captured by the engine as they define the environment in which the component is run. One exception is chown which is rarely set to false and is now always enabled.

• arguments: Merge arguments into argument_groups during a json decode prepare step (PR #574). The --parse_argument_groups option from ns list and config view is deprecated as it is now always enabled.

NEW FUNCTIONALITY

• export json_schema: Add a --strict option to output a subset of the schema representing the internal structure of the Viash config (PR #564).

• config view and ns list: Do not output internal functionality fields (#564). Additionally, add a validation that no internal fields are present when reading a Viash config file.

• project config: Add fields in the project config to specify default values for component config fields (PR #612). This allows for a more DRY approach to defining the same values for multiple components.

MINOR CHANGES

• testbenches: Add testbenches for local dependencies (PR #565).

• testbenches: Refactor testbenches helper functions to uniformize them (PR #565).

• logging: Preserve log order of StdOut and StdErr messages during reading configs in namespaces (PR #571).

• Java 21 support: Update Scala to 2.13.12 and update dependencies (PR #602).

• project config: Output the project config under the default name ProjectConfig instead of Project during schema export (PR #631). This is now important as the project config is now part of the component config. Previously this was overridden as the class name was ViashProject which was less descriptive.

BUG FIXES

• __merge__: Handle invalid yaml during merging (PR #570). There was not enough error handling during this operation. Switched to the more advanced Convert.textToJson helper method.

• config: Anonymize paths in the config when outputting the config (PR #625).