DEP-PIPE enables flexible and scalable deployment and orchestration of Big Data pipelines over the Computing Continuum resources. Through DEP-PIPE, user also monitors the pipeline execution and provides online SLO metrics to the other tools.
DEP-PIPE is based and extends the MAESTRO product of UBITECH. While MAESTRO is clouse source, the components presented below are released open source, while the updates perfomed in the UI are considered important for the exploitation of the DataCloud Toolbox as a whole and remain as closed source extensions of MAESTRO.
- DataCloud Deployment Controlller: https://github.com/DataCloud-project/DEP-PIPE-translator
- Drift Detection Libraries: https://github.com/DataCloud-project/Monitoring-and-Drift-Detection-Libraries
- Security Controller: [https://github.com/DataCloud-project/Monitoring-and-Drift-Detection-Libraries](https://github.com/DataCloud-project/DEP-PIPE-Cluster-Management-and-Protection
DEP-PIPE is used by a)downloading the dedicated container image of the components, or b) by following the provided instructions in order to build the software from code.
Components can be used as standalone services, but for the proper function of DEP-PIPE as part of the DataCloud Toolbox the usage of MAESTRO orchestration is required. Interested users can request (to [email protected]) for using MAESTRO to fully test DEP-PIPE.
Final release of DEP-PIPE released on October 2023.