chore(main): release 8.4.1

[github-actions]

🤖 I have created a release beep boop

8.4.1 (2024-05-14)

⚠ BREAKING CHANGES

• redesigned Snakemake API. It now uses a modern, dataclass based approach (#2403)

Features

• add --sdm short opt for --deployment (#2551) (fd8b4b0)
• add ability to inject conda environments into running Snakefile (#2479) (6140e29)
• add ability to return input functions from input functions. Such nesting is evaluated 10 times at most. Beyond that, an error is thrown. (#2717) (7a47924)
• add cols argument to lookup function; fix various minor bugs on cluster systems (#2651) (ca7a602)
• add flag for marking output as being updated instead of rewritten (update("test.txt")) (#2754) (9ba5d95)
• add function 'exists' for checking the prior existence of files or dirs before workflow execution while considering any remote storage settings. In addition: some bug fixes for error handling and the update/before_update functionality. (ee96393)
• add functionality for deploying sources if no shared FS is assumed (#2486) (76eac3c)
• add method to obtain group args for spawned jobs (bd1b450)
• add option to control software deployment mode (shared or non shared FS) (#2525) (04ec2c0)
• add setting for defining separate local storage prefix for remote jobs; improved ergonomics for semantic helper functions (#2743) (5007e5c)
• Add support for Google Service Accounts and GCE VM network configuration (#2318) (2b754aa)
• add support for Kubernetes service account name spec (#2254) (3370426)
• added localrule directive (#2180) (9c990b0)
• allow config files to be processed with YTE (#2269) (8e1c22f)
• allow default storage provider to be explicitly set to none (#2746) (ce519d7)
• allow detailed configuration of shared FS usage (#2528) (0d34be9)
• allow environment variables in string values of profile (e.g. paths may now contain elements like $USER). (58dc70c)
• allow for more extensive benchmark file in jsonl format (#2691) (de12463)
• allow passing of lists of functions or single functions to expand (#2741) (32e65df)
• allow profiles to be YTE templates; adapt to eido 2.0 (#2325) (67d9ff2)
• allow python expressions in --set-resources (#2521) (022a31e)
• Allow smart_open 7.x (#2745) (d52c1b1)
• Allow the environment variable SNAKEMAKE_CONDA_PREFIX to be present without --use-conda (#2263) (e4eba8d)
• allow to set latency_wait in executor test suite (c0bca0b)
• automatically upload workflow sources to default storage provider if no shared FS is used (a450c49)
• Azbatch executor (#1953) (#2246) (0f9c49f)
• expose ResourceSettings in TestWorkflowsBase (#2770) (e7c323b)
• Faster ci test setup (#2489) (4798e8a)
• implement precommand (#2482) (ff0f979)
• implement semantic helper functions for input and param function handling (#2344) (b4b5e51)
• introduce --workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like --set-threads or --set-resources as multi-level dictionaries in profile config yaml files (#2310) (9675c17)
• limit the number of input/output files in job properties (#2149) (d93f091)
• prepare for storage plugins (#2454) (353a71c)
• redesigned Snakemake API. It now uses a modern, dataclass based approach (#2403) (2be3bfa)
• support for continuously updated input (using Python queues) (#2594) (db1c0ed)
• support for external executor plugins (#2305) (c9eaa4e)
• support for report plugins (#2700) (2f7d4b5)
• Tes auth (#2169) (3326a6f)
• version specific profile config files (profile/config.v8+.yaml with profile/config.yaml as fallback that matches any version) (#2498) (47e5811)

Bug Fixes

• #2130 by patching the protect() method so the path of files in subdirectories is properly resolved during write-protection (#2131) (1a754fd)
• sre_constants import because of deprecation (#2139) (3b326db)
• 2142 log directive in default target rule (#2191) (86e9624)
• adapt linting rule to Python 3.11 (a3a5c58)
• adapt to changes in snakemake-interface-executor-plugins (635c68a)
• add mamba to docker image (eb0c884)
• add missing spaces between lines that get concatenated. (#2268) (7238458)
• add storage provider args to deploy sources command (67178e3)
• add testcase for script directive to work with Python 3.7 and corresponding fix. (0b4ae2e)
• addressing #2197 by allowing 256 character account names in slurm (#2198) (ab58c65)
• allow lookup dpath or query to be a callable (33f1637)
• allow pepfile and pepschema to take pathlib (#2546) (ca91661)
• also inherit rule proxies if there is no rulename modifier specified in a use rule statement (#2440) (1570289)
• always make sure that the original path of source cached files is properly passed into metadata persistence records (#2179) (8bacbd0)
• assume at most 8GB memory for default resources. This way, we avoid exploding memory requirements for large input files that are very unlikely to be put entirely into memory by any tool. (11c2ecc)
• batch bug #2643 (#2650) (2ecb21b)
• better job status queries for slurm executor (#2136) (a4df38c)
• better message about profile usage upon execution (#2391) (cf8aea5)
• binary mem (#2695) (18689b4)
• bug when requesting extended benchmark with slurm (#2855) (0e039ff)
• bug with preemptible rules (#2616) (c6d7141)
• Bump yte from >=1.0,<2.0 to >=1.5.1,<2.0 (#2275) (8c0b34f)
• check template rendering output for leaked input file paths (#2850) (433302e)
• comparison to float in scheduler (ef44d84)
• conda env inside script (#1812) (49cac6a)
• constrain dependencies to match conda experience (#2710) (d9a7a13)
• correctly report lineno (#2584) (967a0d7)
• deduplicate input files before retrieval from storage (#2600) (37cf475)
• detect job paths that leave and then enter a group. Such paths are invalid because then the group depends on itself. (#2527) (5383a4d)
• Detect pandas availability to select serializer (#2300) (e08a771)
• do not allow setting benchmark and between-workflow caching for the same rule. The reason is that when the result is taken from cache, there is no way to fill the benchmark file with any reasonable values. (#2335) (e2d64fa)
• do not distinguish between local and remote rules in dryrun (74b99ec)
• do not overwrite default resources setting in azure batch executor (#2395) (4aef3b9)
• do not pass snakefile as metadata when wms monitor flag is used (#2573) (13b3205)
• do not require cores to be set for non-executing modes (#2646) (30cf026)
• do not require cores to be set for rule-level methods of the workflow API or the corresponding CLI commands (e.g. --lint). (#2629) (2040468)
• Do not scheduler execution message if no jobs are ready (b1c4f47)
• Don't attempt to parse TBDString resources (#2690) (f884b50)
• Enable values with an = sign in default_resources (#2340) (c1c9229)
• ensure lazy evaluation of resource functions/callables (this also entails, for now, a removal of the thread statistics in the yellow job stats table); further, added some clarifying sentences about resource function evaluation to the docs (#2356) (4c591b7)
• ensure that auto deployment of default storage provider works in containers with read-only root home. (1a347ff)
• ensure that log and benchmark files are uploaded to storage as well (#2545) (6aabb5d)
• ensure that targetjob is always forced. This fixes a bug causing run-directive rules to not being executed even when enforced via e.g. -R. (#2448) (b2a60d5)
• ensure user and group rw permissions for metadata files and source cache (#2132) (cc51faa)
• error when detecting mime type during report creation (#2721) (42dad42)
• Escape workdir paths for potential spaces in paths (#2196) (9261f7e)
• expand error on modules with prefix (#1614) (#2803) (c6bfcd7)
• extend workflow test suite by wildcards with slashes (in order to detect bugs that can occur in executors) (#2810) (fc9971b)
• f-string in a more robust style? (#2649) (2a50dc0)
• failure message in cleanup_metadata (#2800) (4c7fe55)
• fix bug causing FileNotFoundError when accessing checkpoint output. (c81954d)
• fix bugs in --summary and --list-input-changes. Removed outdated statement in tutorial. (#2735) (55c06d8)
• fix cache handling and unlock handling (2f4d5e1)
• Fix collect-lookup attribute error (#2687) (e39c74c)
• fix error message for invalid storage provider queries (977951e)
• fix error when passing callable as dpath or query of lookup function (0e5b878)
• fix exception when handling syntax error during parsing (d5a7a56)
• fix false complaints about rules with multiple output files (#2628) (b1b4f5b)
• Fix inconsistencies between detailed summary and normal summary (#2218) (d903123)
• fix index out of bounds error raised by usage of workflow.source_path called from input or params functions (thanks @AKBrueggemann) (#2170) (cf8e6e8)
• fix missed wildcard constraints when using local rule inheritance (#2242) (8e94785)
• fix missing await when opening checkpoint output (#2868) (25a361b)
• fix missing storage information when handling already completed checkpoints. This solves a bug causing failure to retrieve storage files in workflows with checkpoints. (5791c60)
• fix nargs definition for --deploy-sources (fc252c8)
• fix pandas import handling in metadata persistence (27f7b40)
• fix path handling when detective profiles (fe63881)
• fix premature deletion of temp files in combination with checkpoints (#2737) (b22ba5f)
• Fix race condition when creating lock directory (#2225) (66ea4d1)
• fix scrolling behavior in landing page of report for large workflows (63c0c31)
• fix storage handling on windows by converting all paths to posix paths (#2519) (7864a76)
• fix string resource definition in CLI and profile (#2627) (bbd76ae)
• fix wait for files in case of using remote storage and remote execution (#2718) (eec3a5f)
• fixed bug in handling of resource overrides for remote job submission (5c06dd6)
• Fixed plot axis label on report (#2683) (a4c2a03)
• for local execution, always unrestrictedly assume shared FS (#2679) (0bee50b)
• ga4gh executor resources (#2042) (ad6eaef)
• get python version for script environment in a backwards compatible way that works down to python 2.7 (#2161) (44e59b9)
• handle async method in cleanup_storage_objects (#2806) (272205b)
• handle different f-string tokens in py3.12 (#2485) (f2c7613)
• handle non-PEP440 versions of apptainer/singulariy (#2337) (dea6ba8)
• handle storage for local jobs; add test case (6d978ef)
• handling of group jobs when obtaining temp input files (71be1de)
• if report files are within storage, retrieve them from storage before loading into report (60041bd)
• ignore errors when cleaning up runtime cache (#2859) (6df7046)
• import (#2402) (2c831f1)
• improved error handling for storage upload; fixed bugs caused by outdated calls to IOFile.exists(). (720bb84)
• improved error messages in case of invalid storage queries (9671fd0)
• in addition to localrules statement, infer that job is local if it has any input or output file that is marked as local (#2541) (e8b682b)
• invalid extensions with multixt and cache (#1808) (#2823) (a845b1c)
• less frightering message when telling about missing output files as reason for running a job (7b6c9d4)
• limit length of failed logs decorations (#2125) (6fc9243)
• local mtime handling in case of storage plugins and cleaner error message for parallel storage retrieval (#2611) (880b264)
• make checkpoint updates synchronous (#2871) (b0e7ebd)
• Migrate away from deprecated pulp API (#2610) (fb26640)
• migration guide typo and wrong link (#2625) (645f3d1)
• module prefix added twice on expand (#2814) (27416f4)
• move apptainer into separate env in docker image (94e9e2c)
• omit norun jobs when determining remote storage input file retrieval (#2854) (37a7c7f)
• only consider global wildcard_constraints from the same module (#2235) (c412b71)
• only constrain by --max-threads if threads of job are already known (#2790) (5f28fcd)
• only deactivate conda inject envs upon workflow tear down (#2503) (e6dfdd4)
• only download input for local jobs in the main process, not within remote groups (#2842) (97f428b)
• output of rulegraph, closes #2656 (#2671) (f9b9110)
• Panoptes --wms-monitor-arg (#2444) (98d2bdf)
• passing of --set-threads values to remote jobs (#2775) (4fd767a)
• Prevent binary log files to crash snakemake execution with show-failed-logs (#2827) (8a80bda)
• prevents DeprecationWarning caused by using old draft of json schema (#2152) (9791ffb)
• print exceptions when job is not a shell job (#2385) (8a37b85)
• proper interpretation of standard resources given as strings (e.g. runtime as '5m'). Avoid the need to set additional quotes around size or timespan resources. Improved error messages for resource handling. (#2716) (b6636e9)
• proper reuse of rule proxies when importing several times from the same module (#2404) (e867dda)
• properly delete local copies of storage files after remote jobs (#2793) (e3362b0)
• properly handle --touch when using a storage provider (#2705) (fca138d)
• properly resolve wildcards in group components (#2620) (c788a46)
• quote paths given to singularity in order to ensure that it does not fail when paths contain whitespace (#2190) (a572fb7)
• raise error if callable is passed to expand. (#2171) (1f28476)
• remote GS builds too many inventories; io:collect_mtime always uses uncached mtime (#2266) (bad9115)
• remote-azblob-sasToken-Authorization (#1800) (bc854a7)
• remove bash completion entrypoint (no longer supported, was too slow to be usable anyway) (922b53a)
• remove default packages from conda envs (#2749) (027906c)
• remove non-empty local copies of remote storage dirs (#2845) (71b2b87)
• remove superfluous dependency (aad61a0)
• removed distutils from snakemake (#2312) (9b8c362)
• replace pkg_resources for python 3.12 (#2831) (ac144fc)
• report spacing (f3954b3)
• require python >=3.7 again (the python 3.9 dependency was unnecessary) (#2372) (0d0e9c4)
• require toposort >= 1.10 (#2145) (3cb54b8)
• respect APPTAINER_CACHEDIR and allow env variables in --apptainer-prefix and --conda-prefix (#2795) (b1694cd)
• Restore backward compatibility for Google Life Sciences executor (#2461) (5e3a464)
• retrieve files from storage if necessary when calling their open method (e.g. when accessing output files from a checkpoint) (#2839) (5448208)
• retrieve inputs of local rules from storage even if they are intermediate results (#2811) (d158aa8)
• return set instead of list when just --quiet (#2829) (eeb57e2)
• return set of rules when obtaining allowed rules for remote job (2c44cf6)
• rounding for batch calculation (#2064) (cbdbf9b)
• set ignore_incomplete to False in create_conda_envs (#2653) (4834a42)
• Setting the value of ignore_incomplete Fixes #2556 (#2654) (05dac64)
• shadow "full" mode ignore symlinks (#2516) (1d58120)
• show failed logs in executor testcases (92f7bf4)
• show queries of remote storage files instead of local paths in summary (#2860) (ba1db8e)
• single line f-string format error in py3.12 (#2588) (87c06c0)
• Slack log service (#2537) (26eb4ba)
• slurm batch job status queries (#2167) (0bb69e4)
• small typo in error (#2853) (325a715)
• Solve apptainer version issue (#2333) (a876e0f)
• sort report (sub-)categories in lexicographical order (#2449) (d0705ad)
• support list of queries for storage provider (#2674) (d53ef92)
• SyntaxWarning due to invalid escape sequences in non-raw regex pattern string (#2670) (3748d9d)
• SyntaxWarnings due to non-raw regex pattern strings (#2359) (a08c0b0)
• unnecessary set Snakefile in AzBatch executor (#2397) (78e6d6e)
• Update init.py to move "file" param to "print" (#2291) (92352b6)
• update minimum snakemake-interface-storage-plugins version (0ef7226)
• updating of non-dict config values gives error (#2364) (b33aeec)
• Use job.rule.name attribute to fill rule field in summary (#2217) (837c3fd)
• use base64 encoding for passing default resources args to jobs (#2780) (4735bc3)
• use base64 encoding when passing resources and threads to remote jobs (this solves issues with complex quoted resources) (#2778) (a8ee4d8)
• use default container image if nothing is provided (#2677) (109c991)
• use default group settings if not execution workflow (fixes attribute error occurring with --report) (#2617) (21e9964)
• use keyword arguments in _IOFile.open (#2847) (50c84dc)
• use temporary directory (faster, more likely local, always writable) for persistence and source cache in case of remote execution without shared fs (#2502) (c8fa7ba)
• various bug fixes for resource parsing (#2711) (d1daf0b)
• various error handling improvements, fixed logging/error behavior (stdout from dryrun, stderr otherwise) (#2759) (d0d1f48)
• wait for logs before showing them on error (a4ff328)
• when using remote storage: only wait for files if job did not error (8c7ee91)
• wms-monitor now gets data in correct json format (#2347) (7fafa7a)
• wrong rule names when nesting module imports (#1817) (65c79a4)

Performance Improvements

• avoid superflous mtime checks when the same file is referred to by multiple jobs (#2284) (eb6e2e1)
• Gfal2 remote provider using gfal2-python instead of gfal2-utils. (#2128) (0b9bfe5)

Documentation

• add missing doc dependency (7ba9c21)
• add Morten Lund as another maintainer. (293bc05)
• add note on google executor backends (ff8683c)
• add snakemake and reportplugins to req.txt to show all cli options (#2817) (aaa9065)
• added changelog info for >v7.19.1 parsing error of "hh:mm:ss" time format in runtime resource (#2189) (2889f38)
• added Mastodon follow label with just 'Follow' similar to X (#2692) (7e36496)
• api docs polishing (58a6a13)
• basics.rst: suggest VS Code instead of deprecated Atom as IDE (#2368) (1357316)
• Change snakemake-tutorial download link to always be the latest (#2183) (ae8a8f4)
• clarify minimum Snakemake version for profiles (86dc277)
• clarify the channel priority in environment definition deployment.rst (#2352) (76aa964)
• diff 7 and 8 (#2561) (ba22e07)
• document name directive with example (#2534) (cce5551)
• fix #2698 (#2714) (508080b)
• fix a copy&paste (?) mistake (#2386) (d878847)
• fix apidocs theme (1fb5467)
• fix cli options rendering (264c1a9)
• fix explanation on how to use all cores in tutorial (#2733) (6420428)
• fix formatting (087fe63)
• fix heading (1eda36d)
• fix headings (d947f85)
• fix links (420ea9c)
• fix minor typos in a linting rule (#2162) (71e1171)
• fix statement about logging (#2252) (56c24b6)
• fix syntax highlighting in tutorial (2887604)
• fix syntax in cluster example (#2460) (64e9645)
• fix typo (stackoverflow issue) (#2365) (f770984)
• fix typo and link for RO Crate (#2851) (cec0041)
• fix typo in rules.py (#2636) (8bc2919)
• fix typos in --help (#2182) (09f0cbe)
• fixes in migration guide (f8adefa)
• handle overflow of content div (b23e277)
• improve branch function docs (e9d1a11)
• Improve error message when rule contains multiple run/shell/script/notebook/wrapper/template_engine/cwl keywords (#2186) (cd5a3c4)
• mention maintainers (be95e25)
• note on using checkpoint mechanism only for input function, not for params or resources. (#2353) (4be2f9d)
• notes on arm based machines in tutorial docs (0586f04)
• replace nosetest with pytest (#2211) (f6b3c47)
• rust: Fix typo on rust-script version (#2488) (a79dd94)
• update CHANGELOG.md: add minimum Python version bump (#2370) (48e934d)
• update code of conduct email address (3047683)
• update docs for azbatch and dockerhub ref (#2298) (908dbf1)
• update misc/vim/Readme with info for packer.nvim (#2095) (32166a7)
• Update modularization.rst (#2137) (16954c7)
• Update workflow syntax with priority directive (#2188) (af10db5)
• use sphinxawesome-theme instead of lutra (4401e9c)

Miscellaneous Chores

• release 8.4.1 (f8b3bd7)

---

This PR was generated with Release Please. See documentation.