Releases: elyra-ai/elyra
v3.15.0
Quick links
New feature highlights
Pipeline editor: enable specific runtimes
Elyra installations are pre-configured by default to support pipeline execution in local environments (via JupyterLab) and remote environments (Kubeflow Pipelines and Apache Airflow). Beginning with this release you can customize Elyra to only expose runtimes you are planning to use.
Refer to the new Configuring the pipeline editor user guide topic for details.
Kubeflow Pipelines pipeline editor: support for token authentication
Kubeflow Pipelines runtime configurations now support authentication using static bearer tokens. To leverage this authentication mechanism, select EXISTING_BEARER_TOKEN and specify the token value. Elyra passes the specified token as-is to Kubeflow Pipelines.
What's Changed
New Features
- Filter available runtimes based on configuration setting by @kevin-bates in #3114
- Add support for static bearer token authentication for KFP by @ptitzler in #3124
Bug Fixes
- Fix test artifact output location bugs by @ptitzler in #3117
- Hide binder links and references by @ptitzler in #3119
- Determine which kernel to use prior to notebook execution by @ptitzler in #3135
Other
- Require Node.js version 18 (or later) in preparation for version 16 EOL by @ptitzler in #3118
- Update community resources by @ptitzler in #3123
- Lower floor for kfp-tekton dependency by @ptitzler in #3125
Full Changelog: v3.14.3...v3.15.0
Contributors
Assets 2
v3.14.3
What's Changed
Bug Fixes
- Update Kubeflow version information by @ptitzler in #3094
- Fix KFP code generation issues for pipeline parameters by @ptitzler in #3093
- Skip version check for incompatible version strings in
bootstrapper.pyby @cjackal in #3106
Other
- Update github workflows to use v3/v4 actions by @akchinSTC in #3095
- Update function format to conform to black v23 linting by @akchinSTC in #3097
- Remove Python v3.7 from CI tests by @ptitzler in #3098
- Remove references to Python 3.7 by @ptitzler in #3101
- Ensure env stanza is an empty dict if no seed value is given to papermill engine by @kevin-bates in #3103
- Remove references to Jupyterlab 1.x and 2.x from documentation by @ptitzler in #3112
- Bump kfp-tekton to 1.6.2 to pick up kfp 1.8.19 by @ptitzler in #3113
New Contributors
Full Changelog: v3.14.2...v3.14.3
Contributors
Assets 2
v3.14.2
What's Changed
New Features
- Update factory image tags to use SHA256 hashes by @akchinSTC in #3075
Bug Fixes
- Fix write_error function implementation in handlers by @akchinSTC in #3080
- Fix negative parameter behavior by @marthacryan in #3062
- Fix int parameter handling of default zero by @marthacryan in #3090
Other
- Add link to parameters blog post to documentation by @kiersten-stokes in #3064
- Remove caps in dependencies for kfp-related packages by @kevin-bates in #3076
- Disable test due to inconsistent setup in CI by @akchinSTC in #3073
- Decrease jupyter_core floor, cap nbclient for 3.7 by @kevin-bates in #3071
- Update package to include all-optional-dependencies subpackage by @akchinSTC in #3067
- Repackage data files into elyra-server distribution by @akchinSTC in #3081
- Support using podman as container runtime by @akchinSTC in #3082
- Fix oneOf file selection by @marthacryan in #3065
- Update release script for linux environments by @akchinSTC in #3086
Full Changelog: v3.14.1...v3.14.2
Contributors
Assets 2
v3.14.1
What's Changed
Bug Fixes
- Remove jupyter-server-terminal cap and fix release script by @akchinSTC in #3057
Full Changelog: v3.14.0...v3.14.1
Contributors
Assets 2
v3.14.0
Quick links
New feature highlights
Pipeline editor: Configure pipeline parameters for Kubeflow Pipelines
Pipeline parameters allow for customization of pipeline runs and pipeline exports, without to need to modify the pipeline. A pipeline parameter is a typed variable that can be applied to generic or custom nodes. The Kubeflow Pipelines pipeline editor includes a new PIPELINE PARAMETERS tab, where pipeline parameters can be defined.
To make a parameter value available to a generic node, select it from the pipeline parameters list:
Jupyter notebooks and scripts can access the selected parameters and their associated values using environment variables.
To pass a parameter value to a custom node, choose parameter as input and select the desired parameter:
Note that for custom nodes the parameter list only includes parameters that are type-compatible.
You can customize parameter values in the pipeline editor during pipeline submission and pipeline export:
You can also customize parameter values in the pipelines UI in the Kubeflow Central Dashboard.
Pipeline editor: Customize GPU vendor (Kubeflow Pipelines only)
For generic components components you can now specify a custom GPU vendor. Note that the appropriate device plugin must be installed in Kubernetes or node execution will fail. The default vendor remains nvidia.com/gpu.
Pipeline editor: Specify custom pipeline export filename
You can now customize the filename when exporting pipelines. Custom filenames make it easier to retain multiple versions of the same pipeline.
Pipeline editor: Use markdown in comments
You can now use markdown in comments.
Support for Python 3.11
Python 3.11 is now officially supported:
- Elyra can be deployed in Python 3.11 environments
- Generic components can be configured to use runtime images that have Python 3.11 installed
What's Changed
New Features
- Allow for custom export file name by @salonee13 in #2999
- Support config GPU vendor by @typhoonzero in #3029
- Add support for pipeline parameters by @kiersten-stokes and @marthacryan in #3001
Bug Fixes
- Fix code of conduct by @ptitzler in #3027
- Validation: reject binary property input files for custom components by @ptitzler in #3026
- Properly escape string property inputs by @ptitzler in #3048
Other
- Remove broken doc badge and add openssf best practices badge by @ptitzler in #3013
- Update GH pull request template by @akchinSTC in #3015
- Removed unused KeyValueList class by @salonee13 in #3003
- Add reference to latest Elyra blog post to the documentation by @ptitzler in #3011
- Update repository with LFAI onboarding requirements by @akchinSTC in #2980
- Address some code scan issues by @kevin-bates in #3021
- Add and update KFP processor tests by @ptitzler in #3022
- lint Jupyter notebooks by @ptitzler in #3031
- Address warnings produced by tests by @kevin-bates in #3032
- Add Python 3.11 support by @ptitzler in #3033
- Fix generic component schema to unblock frontend changes to file browse feature by @kiersten-stokes in #3040
- Update test dependencies to min of project, address dependabot warnings by @kevin-bates in #3042
- Remove deprecated legacy python package checks by @akchinSTC in #3050
- Add version cap to Jupyterlab by @akchinSTC in #3051
- Export documentation by @salonee13 in #3039
- Add new integration test for testing custom export filename by @kiersten-stokes in #3052
- Cap jupyter-events to <0.5.0 due to jsonschema version conflict by @akchinSTC in #3055
- Add documentation for pipeline parameters by @kiersten-stokes in #3053
- Move build system to use pyproject.toml by @akchinSTC in #3038
New Contributors
- @typhoonzero made their first contribution in #3029
Full Changelog: v3.13.0...v3.14.0
Contributors
Assets 2
v3.13.0
Quick links
New feature highlights
Pipeline editor: customize shared memory size
The amount of shared memory that Kubernetes makes available to pods that execute pipeline nodes can now be customized for individual nodes or all nodes using pipeline defaults.
Pipeline editor: enhanced support for volume mounts
The data volume node property can now be optionally configured to mount a volume in read-only mode (preventing accidental overwrites) and to mount a sub path within the specified volume. Use these new property attributes to impose stricter access control on nodes that should not have unlimited access to mounted data volumes.
Pipeline editor: export Python DSL for Kubeflow Pipelines
The Elyra Pipeline Editor and the elyra-pipeline export CLI command can now export Kubeflow Pipelines Python DSL in addition to the already supported YAML format.
What's Changed
New Features
- Pipeline Editor: Extend data volume node property by @ptitzler in #2961
- Pipeline Editor: Allow for configuration of shared memory size by @ptitzler in #2942
- Document how to refresh component catalog entries by @ptitzler in #2997
- Rewrite KFP code generation by @ptitzler in #2993
Bug Fixes
- Tolerate runtime configuration instances named 'local' by @kevin-bates in #2968
- Fix handling of
cos_object_prefixpipeline property by @kiersten-stokes in #2972 - Skip DisableNodeCaching instance conversion on subsequent attempts by @kiersten-stokes in #2981
- Fix bug with propagation of node caching property by @kiersten-stokes in #3012
Other
- Add 'Review of the latest Elyra Pipeline Editor improvements' blog post by @ptitzler in #2960
- Remove Material UI ThemeProvider and Metadata header by @marthacryan in #2955
- kf-notebook Dockerfile: upgrade Jupyter base image from v1.4 to v1.5 by @ptitzler in #2962
- Prepare to make all Elyra-owned properties object-valued by @kiersten-stokes in #2957
- Fix debugger button state on kernel selection change by @karlaspuldaro in #2977
- Bump pipeline editor dependency to 1.11.0-rc.2 by @ptitzler in #2992
- Add integration tests for Script debugger by @karlaspuldaro in #2971
- Update pipeline editor packages by @akchinSTC in #3009
Full Changelog: v3.12.0...v3.13.0
Contributors
Assets 2
v3.12.0
This release requires pipeline migration.
Quick links
New feature highlights
Pipeline editor: attach Kubernetes labels to nodes
Kubernetes labels can now be defined as pipeline defaults or assigned to individual nodes.
Use labels to assign identifying metadata to the Kubernetes pods that execute the node.
Pipeline editor: disable node caching
Some runtime environments support node output caching, reducing the need to re-execute nodes, which can improve performance and reduce resource usage. For the Kubeflow Pipelines runtime you can now disable node caching for custom nodes that produce output in a non-deterministic way. Node caching can be disabled for all nodes by specifying a pipeline default or for individual nodes.
Generic nodes are currently never cached, due to an Elyra limitation.
Pipeline editor: user-friendly property inputs
Pipeline default properties and node properties can now be configured using custom widgets:
Refer to the documentation for widget descriptions:
- Disable node caching
- Environment variables
- Data volume mounts
- Kubernetes pod labels
- Kubernetes pod annotations
- Kubernetes secrets
- Kubernetes tolerations
Pipeline editor: use files to provide text inputs
In previous releases string-based input properties for custom components could only be provided by entering text or selecting the output of an upstream node. The text input widget in the pipeline editor was extended to also allow for text input from a local file:
In the example above local file url.txt contains a valid URL. The content of this file is not stored in the .pipeline file, only its name and location. The file is only processed by Elyra when you submit or export a pipeline using the pipeline editor or the elyra-pipeline CLI.
The elyra-pipeline describe CLI command was extended to list input files like these as local file dependencies that must be distributed with the .pipeline file:
$ elyra-pipeline describe download.pipeline
...
Pipeline name: download
Description: None specified
Pipeline type: KUBEFLOW_PIPELINES
Pipeline format version: 7.5
Pipeline runtime: Kubeflow Pipelines
Number of generic nodes: 0
Number of custom nodes: 1
Script dependencies: None specified
Notebook dependencies: None specified
Local file dependencies:
- url.txt
...
Improved component catalog connectors
The Apache Airflow package component catalog connector, the Apache Airflow provider package component catalog connector, and the URL component catalog connector now support secured environments where SSL server authenticity can only be validated using certificates based on private public key infrastructure with root and optionally intermediate certificate authorities that are not publicly trusted. Refer to the connector documentation links for details.
What's Changed
New Features
- Support disabling node output caching for custom KFP components by @ptitzler in #2905
- Support not publicly trusted certificates in built-in component catalog connectors by @ptitzler in #2912
- Make multi-valued input fields more user-friendly by @kiersten-stokes in #2927
- Pipeline editor: Allow for configuration of labels by @ptitzler in #2943
Bug Fixes
- Move CLI component cache build to occur before pre-processing by @kiersten-stokes in #2913
- Account for how canvas handles icon size by @ajbozarth in #2914
- Fix rendering of toleration effect in generic components by @kiersten-stokes in #2946
- Fix annotations and label processing by @ptitzler in #2947
Other
- simplify handlers test by @ptitzler in #2900
- Organize doc images by source doc by @ptitzler in #2910
- Improve content of the pipelines documentation topic by @ptitzler in #2909
- Update generic nodes to use static icon api by @ajbozarth in #2908
- Enable debugger on kernel change by @karlaspuldaro in #2903
- Error Dialog Integration Tests by @vhada7 in #2870
- Update Python icon to match JupyterLab by @ajbozarth in #2917
- Add initial support for rjsf in pipeline properties by @marthacryan in #2780
- Add migration section to the pipeline documentation topic by @ptitzler in #2928
- Document how to identify installed Elyra version by @ptitzler in #2930
- Update parse-url dependency due to CVE-2022-2900 by @akchinSTC in #2935
- Fix typo in filename by @ptitzler in #2940
- Explicitly specify units of gigabytes for RAM in Airflow DAG template by @kiersten-stokes in #2949
- Update pipeline editor node package to 1.10 by @akchinSTC in #2950
New Contributors
Full Changelog: v3.11.0...v3.12.0
Contributors
Assets 2
v3.11.1
Quick links
What's Changed
Bug Fixes
- Account for how canvas handles icon size - #2914
- Update generic nodes to use static icon api - #2908
- Enable debugger on kernel change - #2903
Full Changelog: v3.11.0...v3.11.1
v3.11.0
Quick links
New feature highlights
JupyterLab launcher: Find out what's new in Elyra
The JupyterLab launcher now includes a What's new tile in the Elyra category, which links to the release summary of the release you are using, e.g. https://github.com/elyra-ai/elyra/releases/tag/v3.11.0 for Elyra version 3.11. The release summary highlights new features and provides links to release specific resources.
Python code editor: improved debugger integration
This feature is currently experimental. The Elyra Python editor was extended to make it easier to use the JupyterLab debugger. Refer to the user guide for more information.
New Scala code editor
This feature is currently experimental. The Elyra family of editors for JupyterLab now includes a Scala code editor. This editor can also be installed as a stand-alone extension from PyPI.
Note that Scala files are not supported by the Visual Pipeline Editor.
Pipeline editor: support Kubernetes tolerations
The Visual Pipeline Editor now allows for optional input of Kubernetes pod tolerations. Tolerations can be defined as pipeline defaults (applying to all nodes) and for individual nodes and are supported for Kubeflow Pipelines and Apache Airflow.
Pipeline editor: support Kubernetes pod annotations
The Visual Pipeline Editor now allows for optional input of Kubernetes [pod] annotations. Annotations can be defined as pipeline defaults (applying to all nodes) and for individual nodes and are supported for Kubeflow Pipelines and Apache Airflow.
Pipeline editor: remove resource caps for CPU and RAM
In earlier releases the Visual Pipeline Editor capped CPU and RAM resource requests at 99. The caps have been removed.
Component catalog connectors: support loading from local sources
The Apache Airflow package component catalog connector, the Apache Airflow provider package component catalog connector, and the URL component catalog connector, now support locally stored files as source.
Locally stored files can improve the performance of the Visual Pipeline Editor by eliminating the need to download them from remote locations.
Runtime configuration: support for optional public object storage endpoint
Runtime configurations provide Elyra access to external resources, such as Kubeflow Pipelines or Apache Airflow for scalable pipeline execution. In prior releases runtime configurations only allowed for specification of a single URL for cloud object storage. This limitation causes issues in the Elyra UI when object storage deployment policies are configured to enforce separation of read-only and write operations.
To support those deployments it's now possible to optionally configure a read-only endpoint (Public Cloud Object Storage Endpoint) in addition to the existing write endpoint (Cloud Object Storage Endpoint). Refer to the documentation for details.
What's Changed
New Features
- Add support for schema validators by @kevin-bates in #2829
- Add Scala editor by @lresende in #2850
- Add support for Kubernetes annotations by @ptitzler in #2868
- Add support for 'file' URI scheme to URL-based connectors by @ptitzler in #2873
- Add support for Kubernetes tolerations by @ptitzler in #2848
- Makefile: Add capability to validate individual runtime images by @ptitzler in #2879
- Add "what's new" tile to launcher by @salonee13 in #2857
- Add support for COS public endpoint to Kubeflow Pipelines runtime config by @portellaa in #2887
- Add support for COS public endpoint to Airflow runtime config by @ptitzler in #2890
Bug Fixes
- Fix container image build issues for official releases by @ptitzler in #2845
- Fix invalid URL in extension descriptions by @ptitzler in #2860
- Update user guide to enable elyra[all] install on zsh by @leucir in #2839
- Fix Airflow Operator execution bugs in handling of Elyra-owned properties by @kiersten-stokes in #2865
- Fix invalid variable names in error messages by @ptitzler in #2883
- Fix release script and instructions by @ptitzler in #2843
- Fix linting errors in server code by @ptitzler in #2885
- Remove .ONESHELL Makefile directive so multi-step targets properly fail by @kevin-bates in #2888
- Address SVG rendering issues by @ajbozarth in #2895
Other
- Address test warnings by @kevin-bates in #2833
- Add Open Data Hub image refresh instructions by @ptitzler in #2847
- Fix doc requirements to address build issue by @kevin-bates in #2853
- Removes max limit for the GPU and RAM properties by @VNA818-RPI in #2856
- Added development environment dockerfile and requirements file by @salonee13 in #2808
- Remove deprecation warning for local runtime type by @kiersten-stokes in #2862
- Improve 'Running Elyra in an air-gapped environment' documentation topic by @ptitzler in #2871
- Improve repository README by @ptitzler in #2835
- Dev workflow documentation updates by @salonee13 in #2832
- Add export and validation handler tests by @akchinSTC in #2876
- Bump kfp-tekton dependency to 1.3.0 by @ptitzler in #2884
- Add image verification steps to 'Creating a custom runtime container image' documentation by @ptitzler in #2882
- Improve JSON output of 'elyra-pipeline describe' command by @ptitzler in #2878
- Update documentation for elyra development image by @akchinSTC in #2889
- Script editor debugger - experimental by @karlaspuldaro in #2087
- Update release script container builds by @akchinSTC in #2891
- Node package security updates by @akchinSTC in #2896
New Contributors
- @salonee13 made their first contribution in #2808
- @leucir made their first contribution in #2839
- @portellaa made their first contribution in #2887
Full Changelog: v3.10.1...v3.11.0
Contributors
Assets 2
v3.10.1
Quick links
What's Changed
Bug Fixes
- Cap markdown to fix documentation builds by @kiersten-stokes in #2836
Other
- Update typing extensions dependency version cap by @akchinSTC in #2831
Full Changelog: v3.10.0...v3.10.1