diff --git a/README-ZH.md b/README-ZH.md index bcfcfb709..6a1d9cd14 100644 --- a/README-ZH.md +++ b/README-ZH.md @@ -7,7 +7,7 @@ ## 引言 -DataSphere Studio（简称DSS）是微众银行自研的一站式数据应用开发管理门户。 +DataSphere Studio（简称DSS）是微众银行自研的数据应用开发管理集成框架。 基于插拔式的集成框架设计，及计算中间件 [**Linkis**](https://github.com/WeBankFinTech/Linkis) ，可轻松接入上层各种数据应用系统，让数据开发变得简洁又易用。 @@ -37,170 +37,132 @@ DSS主要特点：        3、数据质量管理工具——[Qualitis](https://github.com/WeBankFinTech/Qualitis) -        4、工作流调度工具——[Azkaban](https://azkaban.github.io/) +        4、工作流调度工具——[Schedulis](https://github.com/WeBankFinTech/Schedulis) -        **DSS插拔式的框架设计模式，允许用户快速替换DSS已集成的各个Web系统**。如：将Scriptis替换成Zeppelin，将Azkaban替换成DolphinScheduler。 +        5、数据交换工具——[Exchangis](https://github.com/WeBankFinTech/Exchangis) (**已支持免密跳转，等待Exchangis发版**) + +        6、数据Api服务——[DataApiService](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3/DataApiService%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3.md) + +        7、流式应用开发管理工具——[Streamis](https://github.com/WeBankFinTech/Streamis)(**即将开源**) + +        **DSS插拔式的框架设计模式，允许用户快速替换DSS已集成的各个Web系统**。如：将 Scriptis 替换成Zeppelin，将 Schedulis 替换成DolphinScheduler。 ![DSS一站式](images/zh_CN/readme/onestop.gif) -### 二、基于Linkis计算中间件，打造独有的AppJoint设计理念 +### 二、基于Linkis计算中间件，打造独有的AppConn设计理念 -        AppJoint，是DSS可以简单快速集成各种上层Web系统的核心概念。 +        AppConn，是DSS可以简单快速集成各种上层Web系统的核心概念。 -        AppJoint——应用关节，定义了一套统一的前后台接入规范，可让外部数据应用系统快速简单地接入，成为DSS数据应用开发中的一环。 +        AppConn——应用连接器，定义了一套统一的前后台接入协议，总共分为三级规范，可让外部数据应用系统快速简单地接入，成为DSS数据应用开发中的一环。 + +        AppConn的三级规范即：一级SSO规范，二级组织结构规范，三级开发流程规范； -        DSS通过串联多个AppJoint，编排成一条支持实时执行和定时调度的工作流，用户只需简单拖拽即可完成数据应用的全流程开发。 +        DSS通过串联多个 AppConn，编排成一条支持实时执行和定时调度的工作流，用户只需简单拖拽即可完成数据应用的全流程开发。 -        由于AppJoint对接了Linkis，外部数据应用系统因此具备了资源管控、并发限流、用户资源管理等能力，且允许上下文信息跨系统级共享，彻底告别应用孤岛。 +        由于 AppConn 对接了Linkis，外部数据应用系统因此具备了资源管控、并发限流、用户资源管理等能力，且允许上下文信息跨系统级共享，彻底告别应用孤岛。 -### 三、Project级管理单元 +### 三、Workspace级管理单元 -        以Project为管理单元，组织和管理各数据应用系统的业务应用，定义了一套跨数据应用系统的项目协同开发通用标准。 +        以 Workspace 为管理单元，组织和管理各数据应用系统的业务应用，定义了一套跨数据应用系统的工作空间协同开发通用标准，并提供了用户角色管理能力。 ### 四、已集成的数据应用组件 -        DSS通过实现多个AppJoint，已集成了丰富多样的各种上层数据应用系统，基本可满足用户的数据开发需求。 - -        **用户如果有需要，也可以轻松集成新的数据应用系统，以替换或丰富DSS的数据应用开发流程。** - -        1、DSS的调度能力——Azkaban AppJoint - -           用户的很多数据应用，通常希望具备周期性的调度能力。 - -           目前市面上已有的开源调度系统，与上层的其他数据应用系统整合度低，且难以融通。 - -           DSS通过实现Azkaban AppJoint，允许用户将一个编排好的工作流，一键发布到Azkaban中进行定时调度。 - -           DSS还为调度系统定义了一套标准且通用的DSS工作流解析发布规范，让其他调度系统可以轻松与DSS实现低成本对接。 - -![Azkaban](images/zh_CN/readme/Azkaban_AppJoint.gif) - -        2、数据开发——Scriptis AppJoint - -           什么是[Scriptis](https://github.com/WeBankFinTech/Scriptis)? - -           Scriptis是一款支持在线写SQL、Pyspark、HiveQL等脚本，提交给[Linkis](https://github.com/WeBankFinTech/Linkis)执行的数据分析Web工具，且支持UDF、函数、资源管控和智能诊断等企业级特性。 - -           Scriptis AppJoint为DSS集成了Scriptis的数据开发能力，并允许Scriptis的各种脚本类型，作为DSS工作流的节点，参与到应用开发的流程中。 - -           目前已支持HiveSQL、SparkSQL、Pyspark、Scala等脚本节点类型。 - -![Scriptis](images/zh_CN/readme/Scriptis_AppJoint.gif) - -        3、数据可视化——Visualis AppJoint - -           什么是Visualis? - -           Visualis是一个基于宜信开源项目Davinci二次开发的数据可视化BI工具，为用户在数据安全和权限方面，提供金融级数据可视化能力。 - -           Visualis AppJoint为DSS集成了Visualis的数据可视化能力，并允许数据大屏和仪表盘，作为DSS工作流的节点，与上游的数据集市关联起来。 - -![Visualis](images/zh_CN/readme/Visualis_AppJoint.gif) - -        4、数据质量——Qualitis AppJoint - -           Qualitis AppJoint 为DSS集成数据质量校验能力，将数据质量系统集成到DSS工作流开发中，对数据完整性、正确性等进行校验。 - -![Qualitis](images/zh_CN/readme/Qualitis_AppJoint.gif) - -        5、数据发送——Sender AppJoint - -           Sender AppJoint为DSS集成数据发送能力，目前支持SendEmail节点类型，所有其他节点的结果集，都可以通过邮件发送。 - -           例如：SendEmail节点可直接将Display数据大屏作为邮件发送出来。 - -        6、信号节点——Signal AppJoint - -           EventChecker AppJoint用于强化业务与流程之间的解耦和相互关联。 - -           DataChecker节点：检查库表分区是否存在。 - -           EventSender: 跨工作流和工程的消息发送节点。 - -           EventReceiver: 跨工作流和工程的消息接收节点。 - -        7、功能节点 - -           空节点、子工作流节点。 - -        8、**节点扩展** - -            **根据需要，用户可以简单快速替换DSS已集成的各种功能组件，或新增功能组件。** +        DSS通过实现多个AppConn，已集成了丰富多样的各种上层数据应用系统，基本可满足用户的数据开发需求。 + +        **如果有需要，也可以轻松集成新的数据应用系统，以替换或丰富DSS的数据应用开发流程。** [点我了解如何快速集成新的应用系统](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%BC%80%E5%8F%91%E6%96%87%E6%A1%A3/%E7%AC%AC%E4%B8%89%E6%96%B9%E7%B3%BB%E7%BB%9F%E6%8E%A5%E5%85%A5DSS%E5%BC%80%E5%8F%91%E6%8C%87%E5%8D%97.md) + +| 应用工具 | 描述 | DSS0.X 版本要求 | DSS1.0 版本要求 | 版本规划 | +| --------------- | -------------------------------------------------------------------- | --------------------------------------------------------------------- | ---------- | ------ | +| **DataApiService** | 数据API服务。可快速将SQL脚本发布为一个Restful接口，对外提供Rest访问能力 | 不支持 | >=1.0.0 | 已发布 | +| **Airflow** | 支持将DSS工作流发布到Airflow进行定时调度 | >=0.9.1，尚未合并 | on going | **待规划** | +| **Streamis** | 流式应用开发管理工具。支持发布Flink Jar 和 Flink SQL ，提供流式应用的开发调试和生产管理能力，如：启停、状态监控、checkpoint等。 | 不支持 | >=1.0.0 | **即将发布** | +| **UserManager** | 自动初始化一个DSS新用户所必须的所有用户环境，包含：创建Linux用户、各种用户路径、目录授权等 | >=0.9.1 | on going | **待规划** | +| **EventCheck** | 提供跨业务、跨工程和跨工作流的信号通信能力。 | >=0.5.0 | >=1.0.0 | 已发布 | +| **SendEmail** | 提供数据发送能力，所有其他工作流节点的结果集，都可以通过邮件进行发送 | >=0.5.0 | >=1.0.0 | 已发布 | +| [**Scriptis**](https://github.com/WeBankFinTech/Scriptis) | 支持在线写SQL、Pyspark、HiveQL等脚本，提交给[Linkis](https://github.com/WeBankFinTech/Linkis)执行的数据分析Web工具。 | >=0.5.0 | >=1.0.0 | 已发布 | +| [**Visualis**](https://github.com/WeBankFinTech/Visualis) | 基于宜信开源项目Davinci二次开发的数据可视化BI工具，为用户在数据安全方面提供金融级数据可视化能力。 | >=0.5.0 | >=1.0.0 | 已发布 | +| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis) | 数据质量校验工具，提供数据完整性、正确性等数据校验能力 | >=0.5.0 | >=1.0.0 | **待发布** | +| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | 基于Azkaban二次开发的工作流任务调度系统,具备高性能，高可用和多租户资源隔离等金融级特性。 | >=0.5.0 | >=1.0.0 | 已发布 | +| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | 支持对结构化及无结构化的异构数据源之间的数据传输的数据交换平台 | 不支持 | >=1.0.0 | **待发布** | + ## Demo试用环境 -       由于DataSphereStudio支持执行脚本风险较高，WeDataSphere Demo环境的隔离没有做完，考虑到大家都在咨询Demo环境，决定向社区先定向发放邀请码，接受企业和组织的试用申请。 +       由于 DataSphereStudio 支持执行脚本风险较高，WeDataSphere Demo环境的隔离没有做完，考虑到大家都在咨询Demo环境，决定向社区先定向发放邀请码，接受企业和组织的试用申请。        如果您想试用Demo环境，请加入DataSphere Studio社区用户群（**加群方式请翻到本文档末尾处**），联系团队成员获取邀请码。 -       WeDataSphere Demo环境用户注册页面：https://sandbox.webank.com/wds/dss/#/register +       DataSphereStudio Demo环境用户注册页面：[点我进入](https://www.ozone.space/wds/dss/#/register) -       WeDataSphere Demo环境登录页面：https://sandbox.webank.com/wds/dss/ +       DataSphereStudio Demo环境登录页面：[点我进入](https://www.ozone.space/wds/dss/#/login) -       我们会尽快解决环境隔离问题，争取早日向社区完全开放WeDataSphere Demo环境。 +       **DataSphereStudio1.0 Demo环境将在近期开放，敬请期待**。 -## 与类似系统对比 +## 下载 -        DSS是一个引领数据应用开发管理方向的开源项目，开源社区目前尚没有同类产品。 +        请前往 [DSS releases](https://github.com/WeBankFinTech/DataSphereStudio/releases) 页面下载 DSS 的已编译版本或源码包。 -## 使用场景 +## 编译和安装部署 -        DataSphere Studio适用于以下场景： +请参照 [编译指引](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%BC%80%E5%8F%91%E6%96%87%E6%A1%A3/DSS%E7%BC%96%E8%AF%91%E6%96%87%E6%A1%A3.md) 来编译 DSS 源码。 -        1. 正在筹建或初步具备大数据平台能力，但无任何数据应用工具的场景。 +请参考 [安装部署文档](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/DSS%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md) 来部署 DSS。 -        2. 已具备大数据基础平台能力，且仅有少数数据应用工具的场景。 +## 示例和使用指引 -        3. 已具备大数据基础平台能力，且拥有全部数据应用工具，但工具间尚未打通，用户使用隔离感强、学习成本高的场景。 +请到 [用户使用文档](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3/DSS%E7%94%A8%E6%88%B7%E6%89%8B%E5%86%8C.md) ，了解如何快速使用DSS。 -        4. 已具备大数据基础平台能力，且拥有全部数据应用工具，部分工具已实现对接，但尚未定义统一规范的场景。 +## 文档 -## 快速安装使用 +DSS1.0的完整文档列表，请参见 [DSS-Doc](https://github.com/WeBankFinTech/DataSphereStudio-Doc/tree/main/zh_CN) -点我进入[快速安装使用](docs/zh_CN/ch2/DSS_LINKIS_Quick_Install.md) +以下为 DSS 相关 AppConn 插件的安装指南： -## 架构 +- [DSS的Visualis AppConn插件安装指南](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/VisualisAppConn%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) -![DSS架构](images/zh_CN/readme/architecture.png) +- [DSS的Schedulis AppConn插件安装指南](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/SchedulisAppConn%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) -## 文档列表 +- [DSS的Qualitis AppConn插件安装指南](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/QualitisAppConn%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) -#### 1. 安装编译文档 +- [DSS的Exchangis AppConn插件安装指南](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/ExchangisAppConn%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) -[快速安装使用文档](docs/zh_CN/ch2/DSS_LINKIS_Quick_Install.md) -[**DSS安装常见问题列表**](docs/zh_CN/ch1/DSS安装常见问题列表.md) +## 架构 -[DSS编译文档](docs/zh_CN/ch1/DSS编译文档.md) +![DSS架构](images/zh_CN/readme/architecture.png) -#### 2. 使用文档 -[快速使用文档](docs/zh_CN/ch3/DataSphere_Studio_QuickStart.md) +## 使用场景 -[用户手册](docs/zh_CN/ch3/DSS_User_Manual.md) +        DataSphere Studio适用于以下场景： -#### 3. AppJoint插件安装文档 +        1. 正在筹建或初步具备大数据平台能力，但无任何数据应用工具的场景。 -**以下为手动安装相关插件的指南，DSS一键安装【标准版】已自动安装了以下插件，可忽略。** +        2. 已具备大数据基础平台能力，且仅有少数数据应用工具的场景。 -[DSS的Azkaban AppJoint插件安装指南](docs/zh_CN/ch4/如何接入调度系统Azkaban.md) +        3. 已具备大数据基础平台能力，且拥有全部数据应用工具，但工具间尚未打通，用户使用隔离感强、学习成本高的场景。 -[DSS的Qualitis AppJoint插件安装指南](https://github.com/WeBankFinTech/Qualitis/blob/master/docs/zh_CN/ch1/%E6%8E%A5%E5%85%A5%E5%B7%A5%E4%BD%9C%E6%B5%81%E6%8C%87%E5%8D%97.md) +        4. 已具备大数据基础平台能力，且拥有全部数据应用工具，部分工具已实现对接，但尚未定义统一规范的场景。 -#### 4. 第三方系统如何接入文档 -[DSS如何快速集成第三方系统](docs/zh_CN/ch4/第三方系统接入DSS指南.md) +## 贡献 -#### 5. 架构文档 +我们非常欢迎和期待更多的贡献者参与共建 DSS, 不论是代码、文档，或是其他能够帮助到社区的贡献形式。 -[DSS工程发布到调度系统的架构设计](docs/zh_CN/ch4/DSS工程发布调度系统架构设计.md) +## 联系我们 -更多文档，敬请期待！ +对 DSS 的任何问题和建议，敬请提交issue，以便跟踪处理和经验沉淀共享。 -## 交流贡献 +您也可以扫描下面的二维码，加入我们的微信/QQ群，以获得更快速的响应。 ![交流](images/zh_CN/readme/communication.png) +## 谁在使用 DSS + +我们创建了 [Who is using DSS](https://github.com/WeBankFinTech/DataSphereStudio/issues/1) issue 以便用户反馈和记录谁在使用 DSS，欢迎您注册登记. + +DSS 自2019年开源发布以来，累计已有700多家试验企业和1000+沙盒试验用户，涉及金融、电信、制造、互联网等多个行业。 + ## License DSS is under the Apache 2.0 license. See the [License](LICENSE) file for details. diff --git a/README.md b/README.md index a3bfcfe4e..f37d3639d 100644 --- a/README.md +++ b/README.md @@ -33,19 +33,21 @@ Please be patient, it will take some time to load gif.        c. [Qualitis](https://github.com/WeBankFinTech/Qualitis) - Data Quality Management Tool -        d. [Azkaban](https://azkaban.github.io/) - Batch workflow job scheduler +        d. [Schedulis](https://github.com/WeBankFinTech/Schedulis) - Batch workflow job scheduler + +        f. [Exchangis](https://github.com/WeBankFinTech/Exchangis) - Data Exchange Tool ![DSS one-stop video](images/en_US/readme/onestop.gif) -### 2. AppJoint, based on Linkis，defines a unique design concept +### 2. AppConn, based on Linkis，defines a unique design concept -        AppJoint——application joint, defining unified front-end and back-end +        AppConn——application connector, defining unified front-end and back-end integration specifications, can quickly and easily integrate with external data application systems, making them as part of DSS data application development. -        DSS arranges multiple AppJoints in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations. +        DSS arranges multiple AppConns in series to form a workflow that supports real-time execution and scheduled execution. Users can complete the entire process development of data applications with simple drag and drop operations. -        Since AppJoint is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppJoint also allows sharable context across system level and completely gets away from application silos. +        Since AppConn is integrated with Linkis, the external data application system shares the capabilities of resource management, concurrent limiting, and high performance. AppConn also allows sharable context across system level and completely gets away from application silos. ### 3. Project, as the management unit @@ -53,55 +55,55 @@ Please be patient, it will take some time to load gif. ### 4. Integrated data application components -       a. Azkaban AppJoint —— Batch workflow job scheduler +       a. Schedulis AppConn —— Batch workflow job scheduler          Many data applications developed by users usually require periodic scheduling capability.          At present, the open source scheduling system in the community is pretty unfriendly to integrate with other data application systems. -         DSS implements Azkaban AppJoint, which allows users to publish DSS workflows to Azkaban for regular scheduling. +         DSS implements Schedulis AppConn, which allows users to publish DSS workflows to Azkaban for regular scheduling.          DSS also defines standard and generic workflow parsing and publishing specifications for scheduling systems, allowing other scheduling systems to easily achieve low-cost integration with DSS. ![Azkaban](images/en_US/readme/Azkaban_AppJoint.gif) -       b. Scriptis AppJoint —— Data Development IDE Tool +       b. Scriptis AppConn —— Data Development IDE Tool          What is [Scriptis](https://github.com/WeBankFinTech/Scriptis)?          Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis. -         Scriptis AppJoint integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process. +         Scriptis AppConn integrates the data development capabilities of Scriptis to DSS, and allows various script types of Scriptis to serve as nodes in the DSS workflow to participate in the application development process.          Currently supports HiveSQL, SparkSQL, Pyspark, Scala and other script node types. ![Scriptis](images/en_US/readme/Scriptis_AppJoint.gif) -       c. Visualis AppJoint —— Data Visualization Tool +       c. Visualis AppConn —— Data Visualization Tool          What is [Visualis](https://github.com/WeBankFinTech/Visualis)?          Visualis is a BI tool for data visualization. It provides financial-grade data visualization capabilities on the basis of data security and permissions, based on the open source project Davinci contributed by CreditEase. -         Visualis AppJoint integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market. +         Visualis AppConn integrates data visualization capabilities to DSS, and allows displays and dashboards, as nodes of DSS workflows, to be associated with upstream data market. ![Visualis](images/en_US/readme/Visualis_AppJoint.gif) -       d. Qualitis AppJoint —— Data quality management Tool +       d. Qualitis AppConn —— Data quality management Tool -         Qualitis AppJoint integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow +         Qualitis AppConn integrates data quality verification capabilities for DSS, allows Qualitis as a node in DSS workflow ![Qualitis](images/en_US/readme/Qualitis_AppJoint.gif) -       e. Data Sender——Sender AppJoint +       e. Data Sender——Sender AppConn -         Sender AppJoint provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email. +         Sender AppConn provides data delivery capability for DSS. Currently it supports the SendEmail node type, and the result sets of all other nodes can be sent via email.          For example, the SendEmail node can directly send the screen shot of a display as an email. -       f. Signal AppJoint —— Signal Nodes +       f. Signal AppConn —— Signal Nodes -         Signal AppJoint is used to strengthen the correlation between business and process while keeping them decoupled. +         Signal AppConn is used to strengthen the correlation between business and process while keeping them decoupled.          DataChecker Node：Checks whether a table or partition exists. diff --git a/assembly/bin/appconn-install.sh b/assembly/bin/appconn-install.sh new file mode 100644 index 000000000..a35217b31 --- /dev/null +++ b/assembly/bin/appconn-install.sh @@ -0,0 +1,109 @@ +#!/bin/sh +#Actively load user env +source ~/.bashrc +shellDir=`dirname $0` +workDir=`cd ${shellDir}/..;pwd` + +SOURCE_ROOT=${workDir} + +#load config +source ${SOURCE_ROOT}/conf/config.sh +source ${SOURCE_ROOT}/conf/db.sh + +APPCONN_NAME='' +APPCONN_INSTALL_IP=127.0.0.1 +APPCONN_INSTALL_PORT=8088 + +#echo "Current path of init sql is ${DB_DML_PATH}" +LOCAL_IP="`ifconfig | grep 'inet' | grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $2}'`" + +function isSuccess(){ + if [ $? -ne 0 ]; then + echo "Failed to " + $1 + exit 1 + else + echo "Succeed to" + $1 + fi +} + +PROC_NAME=DSSProjectServerApplication +ProcNumber=`ps -ef |grep -w $PROC_NAME|grep -v grep|wc -l` +if [ $ProcNumber -le 0 ];then + echo "${PROC_NAME} is not running,Please check whether DSS is installed" + exit 1000 +else + echo "Begine to install appconn" +fi + +##choose install mysql mode +function initInstallAppConn() { + echo "Please select the type of installation component?" + echo " 1: schedulis" + echo " 2: visualis" + echo " 3：Your AppConn Name" + echo " 4：exit" + read -p "Please input the choice:" idx + if [[ '1' = "$idx" ]];then + APPCONN_NAME="schedulis" + elif [[ '2' = "$idx" ]];then + APPCONN_NAME="visualis" + elif [[ '4' = "$idx" ]];then + echo "no choice,exit!" + exit 1 + else + APPCONN_NAME=$idx + fi + echo "Current installation component is ${APPCONN_NAME}" + + echo "" + echo "If this machine(127.0.0.1) is installed, enter 1" + echo "For others, you need to enter a complete IP address." + read -p "Please enter the ip of appconn: " ip + APPCONN_INSTALL_IP=$ip + if [[ '1' = "$ip" ]];then + APPCONN_INSTALL_IP="127.0.0.1" + fi + echo "You input ip is ${APPCONN_INSTALL_IP}" + + echo "" + read -p "Please enter the port of appconn:" port + APPCONN_INSTALL_PORT=$port + echo "You input ip is ${APPCONN_INSTALL_PORT}" +} + +function replaceCommonIp() { + if [[ $APPCONN_INSTALL_IP == "127.0.0.1" ]] || [[ $APPCONN_INSTALL_IP == "0.0.0.0" ]];then + echo "APPCONN_INSTALL_IP is equals $APPCONN_INSTALL_IP, we will change it to ip address" + APPCONN_INSTALL_IP=$LOCAL_IP + fi +} + +##choose execute mysql mode +function executeSQL() { + TEMP_DB_DML_PATH=${SOURCE_ROOT}/dss-appconns/${APPCONN_NAME}/db + DB_DML_PATH=$TEMP_DB_DML_PATH/init_real.sql + cp -rf $TEMP_DB_DML_PATH/init.sql $DB_DML_PATH + sed -i "s/APPCONN_INSTALL_IP/$APPCONN_INSTALL_IP/g" $DB_DML_PATH + sed -i "s/APPCONN_INSTALL_PORT/$APPCONN_INSTALL_PORT/g" $DB_DML_PATH + sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH + mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH" + isSuccess "source $DB_DML_PATH" + echo "the table update finished" +} + +echo "" +echo "step1:Initialize installation settings" +initInstallAppConn +echo "" + +echo "step2:replaceIp" +replaceCommonIp +echo "" + +echo "step3:update database" +executeSQL +echo "" + +echo "step4:refresh appconn load" +curl -H "Token-Code:BML-AUTH" -H "Token-User:hadoop" -X GET http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}/api/rest_j/v1/dss/framework/project/appconn/${APPCONN_NAME}/load +echo "" \ No newline at end of file diff --git a/bin/checkEnv.sh b/assembly/bin/checkEnv.sh similarity index 97% rename from bin/checkEnv.sh rename to assembly/bin/checkEnv.sh index d51bd5ca2..39f546d26 100644 --- a/bin/checkEnv.sh +++ b/assembly/bin/checkEnv.sh @@ -36,8 +36,6 @@ echo "<-----start to check used cmd---->" need_cmd yum need_cmd java need_cmd mysql -need_cmd unzip -need_cmd expect need_cmd telnet need_cmd tar need_cmd sed diff --git a/assembly/bin/excecuteSQL.sh b/assembly/bin/excecuteSQL.sh new file mode 100644 index 000000000..8896cf3c0 --- /dev/null +++ b/assembly/bin/excecuteSQL.sh @@ -0,0 +1,102 @@ +#!/bin/sh + +function checkExternalServer(){ + echo "telnet check for your $SERVER_NAME, if you wait for a long time,may be your $SERVER_NAME does not prepared" + result=`echo -e "\n" | telnet $EXTERNAL_SERVER_IP $EXTERNAL_SERVER_PORT 2>/dev/null | grep Connected | wc -l` + if [ $result -eq 1 ]; then + echo "$SERVER_NAME is OK." + else + echo "$SERVER_NAME is Bad. You need to prepare the' $SERVER_NAME ' environment in advance" + exit 1 + fi +} + +## choose install mode +function chooseInstallMode() { + echo "Simple installation mode" + #check for Java + checkJava + #check for mysql + SERVER_NAME=MYSQL + EXTERNAL_SERVER_IP=$MYSQL_HOST + EXTERNAL_SERVER_PORT=$MYSQL_PORT + checkExternalServer +} + +##choose install mysql mode +function chooseInstallMySQLMode() { + echo "Do you want to clear Dss table information in the database?" + echo " 1: Do not execute table-building statements" + echo " 2: Dangerous! Clear all data and rebuild the tables." + echo "" + MYSQL_INSTALL_MODE=1 + read -p "Please input the choice:" idx + if [[ '2' = "$idx" ]];then + MYSQL_INSTALL_MODE=2 + echo "You chose Rebuild the table" + elif [[ '1' = "$idx" ]];then + MYSQL_INSTALL_MODE=1 + echo "You chose not execute table-building statements" + else + echo "no choice,exit!" + exit 1 + fi + + ##init db + if [[ '2' = "$MYSQL_INSTALL_MODE" ]];then + ENV_FLAG="dev" + DB_CONF_PATH=${workDir}/db + DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql + replaceAppConnInstanceSQL + executeSQL + fi +} + +##choose execute mysql mode +function executeSQL() { + chooseInstallMode + + sed -i "s/GATEWAY_INSTALL_IP/$GATEWAY_INSTALL_IP/g" $DB_DML_PATH + sed -i "s/GATEWAY_PORT/$GATEWAY_PORT/g" $DB_DML_PATH + + sed -i "s#DSS_INSTALL_HOME_VAL#$DSS_INSTALL_HOME#g" $DB_DML_PATH + + mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_CONF_PATH/dss_ddl.sql" + isSuccess "source dss_ddl.sql" + mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source $DB_DML_PATH" + isSuccess "source dss_dml_real.sql" + echo "Rebuild the table" +} + +function replaceAppConnInstanceSQL() { + DB_DML_PATH=$DB_CONF_PATH/dss_dml_real.sql + cp -rf $DB_CONF_PATH/dss_dml.sql $DB_DML_PATH + sed -i "s#ORCHESTRATOR_IP#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP#g" $DB_DML_PATH + sed -i "s#ORCHESTRATOR_PORT#$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT#g" $DB_DML_PATH + + sed -i "s#WORKFLOW_IP#$DSS_WORKFLOW_SERVER_INSTALL_IP#g" $DB_DML_PATH + sed -i "s#WORKFLOW_PORT#$DSS_WORKFLOW_SERVER_PORT#g" $DB_DML_PATH + + sed -i "s#EVENTCHECKER_JDBC_URL#$EVENTCHECKER_JDBC_URL#g" $DB_DML_PATH + sed -i "s#EVENTCHECKER_JDBC_USERNAME#$EVENTCHECKER_JDBC_USERNAME#g" $DB_DML_PATH + sed -i "s#EVENTCHECKER_JDBC_PASSWORD#$EVENTCHECKER_JDBC_PASSWORD#g" $DB_DML_PATH + + sed -i "s#DATACHECKER_JOB_JDBC_URL#$DATACHECKER_JOB_JDBC_URL#g" $DB_DML_PATH + sed -i "s#DATACHECKER_JOB_JDBC_USERNAME#$DATACHECKER_JOB_JDBC_USERNAME#g" $DB_DML_PATH + sed -i "s#DATACHECKER_JOB_JDBC_PASSWORD#$DATACHECKER_JOB_JDBC_PASSWORD#g" $DB_DML_PATH + + sed -i "s#DATACHECKER_BDP_JDBC_URL#$DATACHECKER_BDP_JDBC_URL#g" $DB_DML_PATH + sed -i "s#DATACHECKER_BDP_JDBC_USERNAME#$DATACHECKER_BDP_JDBC_USERNAME#g" $DB_DML_PATH + sed -i "s#DATACHECKER_BDP_JDBC_PASSWORD#$DATACHECKER_BDP_JDBC_PASSWORD#g" $DB_DML_PATH + + sed -i "s#BDP_MASK_IP#127.0.0.1#g" $DB_DML_PATH + sed -i "s#BDP_MASK_PORT#8087#g" $DB_DML_PATH + + sed -i "s#EMAIL_HOST#${EMAIL_HOST}#g" $DB_DML_PATH + sed -i "s#EMAIL_PORT#${EMAIL_PORT}#g" $DB_DML_PATH + sed -i "s#EMAIL_USERNAME#${EMAIL_USERNAME}#g" $DB_DML_PATH + sed -i "s#EMAIL_PASSWORD#${EMAIL_PASSWORD}#g" $DB_DML_PATH + sed -i "s#EMAIL_PROTOCOL#${EMAIL_PROTOCOL}#g" $DB_DML_PATH +} + +chooseInstallMySQLMode diff --git a/assembly/bin/install.sh b/assembly/bin/install.sh new file mode 100644 index 000000000..b02b48c0a --- /dev/null +++ b/assembly/bin/install.sh @@ -0,0 +1,313 @@ +#!/bin/sh +#Actively load user env +if [ -f "~/.bashrc" ];then + echo "Warning! user bashrc file does not exist." +else + source ~/.bashrc +fi + +shellDir=`dirname $0` +workDir=`cd ${shellDir}/..;pwd` + +SERVER_IP="" +SERVER_HOME="" + +local_host="`hostname --fqdn`" +LOCAL_IP="`ifconfig | grep 'inet' | grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $2}'`" + +#To be compatible with MacOS and Linux +txt="" +if [[ "$OSTYPE" == "darwin"* ]]; then + txt="''" +elif [[ "$OSTYPE" == "linux-gnu" ]]; then + # linux + txt="" +elif [[ "$OSTYPE" == "cygwin" ]]; then + echo "dss not support Windows operating system" + exit 1 +elif [[ "$OSTYPE" == "msys" ]]; then + echo "dss not support Windows operating system" + exit 1 +elif [[ "$OSTYPE" == "win32" ]]; then + echo "dss not support Windows operating system" + exit 1 +elif [[ "$OSTYPE" == "freebsd"* ]]; then + txt="" +else + echo "Operating system unknown, please tell us(submit issue) for better service" + exit 1 +fi + +function isSuccess(){ + if [ $? -ne 0 ]; then + echo "Failed to " + $1 + exit 1 + else + echo "Succeed to" + $1 + fi +} + +function checkJava(){ + java -version + isSuccess "execute java --version" +} + +checkJava + +dos2unix ${workDir}/config/* +dos2unix ${workDir}/bin/* + +echo "step1:load config" +source ${workDir}/config/config.sh +source ${workDir}/config/db.sh + +DSS_FILE_PATH="$workDir/$DSS_FILE_NAME" +#dos2unix ${DSS_FILE_PATH}/sbin/* +#dos2unix ${DSS_FILE_PATH}/sbin/ext/* +if [ -z $DSS_FILE_NAME ]; then + echo "DSS_FILE_NAME is null " + exit 1 +fi + +function replaceCommonIp() { + if [ -z "$DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP" ]; then + DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP=$LOCAL_IP + fi + if [ -z "$DSS_FRAMEWORK_PROJECT_SERVER_PORT" ]; then + DSS_FRAMEWORK_PROJECT_SERVER_PORT=9002 + fi + + if [ -z "$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP" ]; then + DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP=$LOCAL_IP + fi + if [ -z "$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT" ]; then + DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT=9003 + fi + + if [ -z "$DSS_APISERVICE_SERVER_INSTALL_IP" ]; then + DSS_APISERVICE_SERVER_INSTALL_IP=$LOCAL_IP + fi + if [ -z "$DSS_APISERVICE_SERVER_PORT" ]; then + DSS_APISERVICE_SERVER_PORT=9004 + fi + + if [ -z "$DSS_WORKFLOW_SERVER_INSTALL_IP" ]; then + DSS_WORKFLOW_SERVER_INSTALL_IP=$LOCAL_IP + fi + if [ -z "$DSS_WORKFLOW_SERVER_PORT" ]; then + DSS_WORKFLOW_SERVER_PORT=9005 + fi + + if [ -z "$DSS_FLOW_EXECUTION_SERVER_INSTALL_IP" ]; then + DSS_FLOW_EXECUTION_SERVER_INSTALL_IP=$LOCAL_IP + fi + if [ -z "$DSS_FLOW_EXECUTION_SERVER_PORT" ]; then + DSS_FLOW_EXECUTION_SERVER_PORT=9006 + fi + + if [ -z "$DSS_DATAPIPE_SERVER_INSTALL_IP" ]; then + DSS_DATAPIPE_SERVER_INSTALL_IP=$LOCAL_IP + fi + if [ -z "$DSS_DATAPIPE_SERVER_PORT" ]; then + DSS_DATAPIPE_SERVER_PORT=9008 + fi + + if [[ $GATEWAY_INSTALL_IP == "127.0.0.1" ]] || [ -z "$GATEWAY_INSTALL_IP" ]; then + echo "GATEWAY_INSTALL_IP is equals $GATEWAY_INSTALL_IP ,we will change it to ip address" + GATEWAY_INSTALL_IP=$LOCAL_IP + fi + if [[ $EUREKA_INSTALL_IP == "127.0.0.1" ]] || [ -z "$EUREKA_INSTALL_IP" ]; then + echo "EUREKA_INSTALL_IP is equals $EUREKA_INSTALL_IP ,we will change it to ip address" + EUREKA_INSTALL_IP=$LOCAL_IP + fi +} +##提换真实的IP +replaceCommonIp + +EUREKA_URL=http://$EUREKA_INSTALL_IP:$EUREKA_PORT/eureka/ + +## excecute sql +source ${workDir}/bin/excecuteSQL.sh + +function changeCommonConf(){ + sed -i "s#defaultZone:.*#defaultZone: $EUREKA_URL#g" $CONF_APPLICATION_YML + sed -i "s#hostname:.*#hostname: $SERVER_IP#g" $CONF_APPLICATION_YML + sed -i "s#wds.linkis.server.mybatis.datasource.url.*#wds.linkis.server.mybatis.datasource.url=jdbc:mysql://${MYSQL_HOST}:${MYSQL_PORT}/${MYSQL_DB}?characterEncoding=UTF-8#g" $CONF_DSS_PROPERTIES + sed -i "s#wds.linkis.server.mybatis.datasource.username.*#wds.linkis.server.mybatis.datasource.username=$MYSQL_USER#g" $CONF_DSS_PROPERTIES + sed -i "s#wds.linkis.server.mybatis.datasource.password.*#***REMOVED***$MYSQL_PASSWORD#g" $CONF_DSS_PROPERTIES + sed -i "s#wds.linkis.gateway.ip.*#wds.linkis.gateway.ip=$GATEWAY_INSTALL_IP#g" $CONF_DSS_PROPERTIES + sed -i "s#wds.linkis.gateway.port.*#wds.linkis.gateway.port=$GATEWAY_PORT#g" $CONF_DSS_PROPERTIES + sed -i "s#wds.linkis.gateway.url.*#wds.linkis.gateway.url=http://$GATEWAY_INSTALL_IP:$GATEWAY_PORT/#g" $CONF_DSS_PROPERTIES + sed -i "s#wds.linkis.gateway.wtss.url.*#wds.linkis.gateway.wtss.url=http://$GATEWAY_INSTALL_IP:$GATEWAY_PORT/#g" $CONF_DSS_PROPERTIES +} + +##function start +function changeConf(){ + sed -i "s#spring.server.port=.*#spring.server.port=$SERVER_PORT#g" $CONF_SERVER_PROPERTIES + if [[ $SERVER_NAME == "dss-framework-orchestrator-server" ]] || [[ $SERVER_NAME == "dss-workflow-server" ]]; then + SERVER_FULL_NAME=$SERVER_NAME + SERVER_FULL_NAME=$SERVER_NAME-$ENV_FLAG + sed -i "s#spring.spring.application.name=.*#spring.spring.application.name=$SERVER_FULL_NAME#g" $CONF_SERVER_PROPERTIES + fi + sed -i "s#wds.dss.appconn.scheduler.project.store.dir.*#wds.dss.appconn.scheduler.project.store.dir=$WDS_SCHEDULER_PATH#g" $CONF_SERVER_PROPERTIES + isSuccess "subsitution $CONF_SERVER_PROPERTIES of $SERVER_NAME" +} +##function end + + +UPLOAD_PUBLIC_IPS="" +##function start +function uploadProjectFile(){ + if [[ $SERVER_IP == "127.0.0.1" ]]; then + SERVER_IP=$local_host + fi + #echo "$SERVER_NAME-step3:copy install package" + # upload project conf + # cp -rfp $SSH_PORT ${workDir}/config/{$SERVER_NAME}.properties $CONF_PATH + if [[ $UPLOAD_PUBLIC_IPS == *",${ENV_FLAG}-$SERVER_IP,"* ]]; then + return 0 + fi + cp -rfp ${DSS_FILE_PATH}/* $SERVER_HOME + cp -rfp ${workDir}/bin $SERVER_HOME + cp -rfp ${workDir}/config/* $SERVER_HOME/conf + sudo chown -R $deployUser:$deployUser $SERVER_HOME + UPLOAD_PUBLIC_IPS="$UPLOAD_PUBLIC_IPS,${ENV_FLAG}-$SERVER_IP," + changeCommonConf +# echo "UPLOAD_PUBLIC_IPS-->$UPLOAD_PUBLIC_IPS" +} + +##function start +function installPackage(){ + if [[ $SERVER_IP == "127.0.0.1" ]]; then + SERVER_IP=$local_host + fi + if [ -z $SERVER_NAME ]; then + echo "ERROR:SERVER_NAME is null " + exit 1 + fi + uploadProjectFile + # change configuration + changeConf +} + +function dssWebInstall(){ +if ! test -e ${LINKIS_DSS_HOME}/wedatasphere-dss-web*.zip; then + echo "**********Error: please put wedatasphere-dss-web-xxx.zip in ${LINKIS_DSS_HOME}! " + exit 1 +else + echo "Start to unzip dss web package." + unzip -d ${LINKIS_DSS_HOME}/web/ -o ${LINKIS_DSS_HOME}/wedatasphere-dss-web-*.zip > /dev/null 2>&1 + sed -i "s#linkis_url.*#linkis_url=${LINKIS_GATEWAY_URL}#g" ${LINKIS_DSS_HOME}/web/config.sh + isSuccess "Unzip dss web package to ${LINKIS_DSS_HOME}/web" +fi +} + +##Install dss projects +function installDssProject() { +# if [ "$DSS_INSTALL_HOME" != "" ] +# then +# rm -rf $DSS_INSTALL_HOME +# fi + echo "" + echo "-----------------DSS install start--------------------" + SERVER_HOME=$DSS_INSTALL_HOME + if [ "$SERVER_HOME" == "" ] + then + export SERVER_HOME=${workDir}/DssInstall + fi + if [ -d $SERVER_HOME ] && [ "$SERVER_HOME" != "$workDir" ]; then + rm -r $SERVER_HOME-bak + echo "mv $SERVER_HOME $SERVER_HOME-bak" + mv $SERVER_HOME $SERVER_HOME-bak + fi + echo "create dir SERVER_HOME: $SERVER_HOME" + sudo mkdir -p $SERVER_HOME;sudo chown -R $deployUser:$deployUser $SERVER_HOME + isSuccess "Create the dir of $SERVER_HOME" + + echo "" + SERVER_NAME=dss-framework-project-server + SERVER_IP=$DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP + SERVER_PORT=$DSS_FRAMEWORK_PROJECT_SERVER_PORT + + UPLOAD_LIB_FILES=$DSS_FILE_PATH/lib/dss-framework/$SERVER_NAME + LIB_PATH=$SERVER_HOME/lib/dss-framework + LOG_PATH=$SERVER_HOME/logs/dss-framework/$SERVER_NAME + CONF_SERVER_PROPERTIES=$SERVER_HOME/conf/$SERVER_NAME.properties + CONF_DSS_PROPERTIES=$SERVER_HOME/conf/dss.properties + CONF_APPLICATION_YML=$SERVER_HOME/conf/application-dss.yml + ###install project-Server + installPackage + echo "" + + SERVER_NAME=dss-framework-orchestrator-server + SERVER_IP=$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP + SERVER_PORT=$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT + UPLOAD_LIB_FILES=$DSS_FILE_PATH/lib/dss-framework/$SERVER_NAME + LIB_PATH=$SERVER_HOME/lib/dss-framework + LOG_PATH=$SERVER_HOME/logs/dss-framework/$SERVER_NAME + CONF_SERVER_PROPERTIES=$SERVER_HOME/conf/$SERVER_NAME.properties + CONF_DSS_PROPERTIES=$SERVER_HOME/conf/dss.properties + CONF_APPLICATION_YML=$SERVER_HOME/conf/application-dss.yml + ###install project-Server + installPackage + echo "" + + SERVER_NAME=dss-apiservice-server + SERVER_IP=$DSS_APISERVICE_SERVER_INSTALL_IP + SERVER_PORT=$DSS_APISERVICE_SERVER_PORT + UPLOAD_LIB_FILES=$DSS_FILE_PATH/lib/dss-apps/$SERVER_NAME + LIB_PATH=$SERVER_HOME/lib/dss-apps + LOG_PATH=$SERVER_HOME/logs/dss-apps/$SERVER_NAME + CONF_SERVER_PROPERTIES=$SERVER_HOME/conf/$SERVER_NAME.properties + CONF_DSS_PROPERTIES=$SERVER_HOME/conf/dss.properties + CONF_APPLICATION_YML=$SERVER_HOME/conf/application-dss.yml + ###install dss-apiservice-server + installPackage + echo "" + + SERVER_NAME=dss-datapipe-server + SERVER_IP=$DSS_DATAPIPE_SERVER_INSTALL_IP + SERVER_PORT=$DSS_DATAPIPE_SERVER_PORT + UPLOAD_LIB_FILES=$DSS_FILE_PATH/lib/dss-apps/$SERVER_NAME + LIB_PATH=$SERVER_HOME/lib/dss-apps + LOG_PATH=$SERVER_HOME/logs/dss-apps/$SERVER_NAME + CONF_SERVER_PROPERTIES=$SERVER_HOME/conf/$SERVER_NAME.properties + CONF_DSS_PROPERTIES=$SERVER_HOME/conf/dss.properties + CONF_APPLICATION_YML=$SERVER_HOME/conf/application-dss.yml + ###install dss-datapipe-server + installPackage + echo "" + + ##Flow execution Install + PACKAGE_DIR=dss + SERVER_NAME=dss-flow-execution-server + SERVER_IP=$DSS_FLOW_EXECUTION_SERVER_INSTALL_IP + SERVER_PORT=$DSS_FLOW_EXECUTION_SERVER_PORT + UPLOAD_LIB_FILES=$DSS_FILE_PATH/lib/dss-orchestrator/$SERVER_NAME + LIB_PATH=$SERVER_HOME/lib/dss-orchestrator + LOG_PATH=$SERVER_HOME/logs/dss-orchestrator/$SERVER_NAME + CONF_SERVER_PROPERTIES=$SERVER_HOME/conf/$SERVER_NAME.properties + CONF_DSS_PROPERTIES=$SERVER_HOME/conf/dss.properties + CONF_APPLICATION_YML=$SERVER_HOME/conf/application-dss.yml + ###Install flow execution + installPackage + echo "" + + SERVER_NAME=dss-workflow-server + SERVER_IP=$DSS_WORKFLOW_SERVER_INSTALL_IP + SERVER_PORT=$DSS_WORKFLOW_SERVER_PORT + UPLOAD_LIB_FILES=$DSS_FILE_PATH/lib/dss-orchestrator/$SERVER_NAME + LIB_PATH=$SERVER_HOME/lib/dss-orchestrator + LOG_PATH=$SERVER_HOME/logs/dss-orchestrator/$SERVER_NAME + CONF_SERVER_PROPERTIES=$SERVER_HOME/conf/$SERVER_NAME.properties + CONF_DSS_PROPERTIES=$SERVER_HOME/conf/dss.properties + CONF_APPLICATION_YML=$SERVER_HOME/conf/application-dss.yml + ###install dss-workflow-server + installPackage + echo "" + echo "-----------------DSS install end--------------------" + echo "" + +} +ENV_FLAG="dev" +installDssProject \ No newline at end of file diff --git a/assembly/config/config.sh b/assembly/config/config.sh new file mode 100644 index 000000000..ba5307eac --- /dev/null +++ b/assembly/config/config.sh @@ -0,0 +1,81 @@ +### deploy user +deployUser=hadoop + +##微服务的最大内存使用量 +SERVER_HEAP_SIZE="512M" + + +### The install home path of DSS，Must provided +DSS_INSTALL_HOME=/appcom/Install/dss-dev + +### Linkis EUREKA information. # Microservices Service Registration Discovery Center +EUREKA_INSTALL_IP=127.0.0.1 +EUREKA_PORT=20303 + +### Specifies the user workspace, which is used to store the user's script files and log files. +### Generally local directory +#WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/ +#### Path to store job ResultSet：file or hdfs path +#RESULT_SET_ROOT_PATH=hdfs:///tmp/linkis + +### Linkis Gateway information +GATEWAY_INSTALL_IP=127.0.0.1 +GATEWAY_PORT=9001 + +#for azkaban +WDS_SCHEDULER_PATH=file:///appcom/tmp/wds/scheduler + +################### The install Configuration of all Micro-Services ##################### +# +# NOTICE: +# 1. If you just wanna try, the following micro-service configuration can be set without any settings. +# These services will be installed by default on this machine. +# 2. In order to get the most complete enterprise-level features, we strongly recommend that you install +# the following microservice parameters +# + +### DSS_SERVER +### This service is used to provide dss-server capability. + +### project-server +DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP=127.0.0.1 +DSS_FRAMEWORK_PROJECT_SERVER_PORT=9002 +### orchestrator-server +DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP=127.0.0.1 +DSS_FRAMEWORK_ORCHESTRATOR_SERVER_PORT=9003 +### apiservice-server +DSS_APISERVICE_SERVER_INSTALL_IP=127.0.0.1 +DSS_APISERVICE_SERVER_PORT=9004 +### dss-workflow-server +DSS_WORKFLOW_SERVER_INSTALL_IP=127.0.0.1 +DSS_WORKFLOW_SERVER_PORT=9005 +### dss-flow-execution-server +DSS_FLOW_EXECUTION_SERVER_INSTALL_IP=127.0.0.1 +DSS_FLOW_EXECUTION_SERVER_PORT=9006 +###dss-datapipe-server +DSS_DATAPIPE_SERVER_INSTALL_IP=127.0.0.1 +DSS_DATAPIPE_SERVER_PORT=9008 + +############## ############## dss_appconn_instance configuration start ############## ############## +EVENTCHECKER_JDBC_URL="jdbc:mysql://127.0.0.1:3306/dss_linkis?characterEncoding=UTF-8" +EVENTCHECKER_JDBC_USERNAME=hadoop +EVENTCHECKER_JDBC_PASSWORD=hadoop + +DATACHECKER_JOB_JDBC_URL="jdbc:mysql://127.0.0.1:3306/hive_gz_bdap_test_01?useUnicode=true" +DATACHECKER_JOB_JDBC_USERNAME=hadoop +DATACHECKER_JOB_JDBC_PASSWORD=hadoop + +DATACHECKER_BDP_JDBC_URL="jdbc:mysql://127.0.0.1:3306/uat2_metastore?characterEncoding=UTF-8" +DATACHECKER_BDP_JDBC_USERNAME=hadoop +DATACHECKER_BDP_JDBC_PASSWORD=hadoop + +EMAIL_HOST=smtp.163.com +EMAIL_PORT=25 +EMAIL_USERNAME=xxx@163.com +EMAIL_PASSWORD=xxxxx +EMAIL_PROTOCOL=smtp +############## ############## dss_appconn_instance configuration end ############## ############## + +DSS_VERSION=1.0.0 + +DSS_FILE_NAME="dss-$DSS_VERSION" \ No newline at end of file diff --git a/assembly/config/db.sh b/assembly/config/db.sh new file mode 100644 index 000000000..148ae93a9 --- /dev/null +++ b/assembly/config/db.sh @@ -0,0 +1,8 @@ +### for DSS-Server and Eventchecker APPCONN +MYSQL_HOST=127.0.0.1 +MYSQL_PORT=3306 +MYSQL_DB=dss_dev +MYSQL_USER=hadoop +MYSQL_PASSWORD=hadoop + + diff --git a/assembly/dss-package/pom.xml b/assembly/dss-package/pom.xml new file mode 100644 index 000000000..59643ab2c --- /dev/null +++ b/assembly/dss-package/pom.xml @@ -0,0 +1,179 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + dss-package + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + + + jackson-databind + com.fasterxml.jackson.core + + + jackson-core + com.fasterxml.jackson.core + + + jackson-core-asl + org.codehaus.jackson + + + httpclient + org.apache.httpcomponents + + + hibernate-validator + org.hibernate.validator + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + + + jackson-core + com.fasterxml.jackson.core + + + + + com.webank.wedatasphere.dss + dss-sender-service + ${dss.version} + + + + org.apache.httpcomponents + httpclient + ${httpclient.version} + + + + org.apache.httpcomponents + httpmime + ${httpmime.version} + + + commons-beanutils + commons-beanutils + ${beanutils.version} + + + + com.webank.wedatasphere.linkis + linkis-hadoop-common + ${linkis.version} + + + com.fasterxml.jackson.core + jackson-databind + ${fasterxml.jackson.version} + + + + org.apache.commons + commons-math3 + ${math3.version} + + + + org.apache.commons + commons-lang3 + ${commons.lang3.version} + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + + + + org.glassfish.jersey.ext + jersey-bean-validation + 2.21 + + + + + + + + org.apache.maven.plugins + maven-install-plugin + + true + + + + org.apache.maven.plugins + maven-antrun-plugin + + + package + + run + + + + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + + + dist + package + + single + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + + + \ No newline at end of file diff --git a/assembly/dss-package/src/main/assembly/distribution.xml b/assembly/dss-package/src/main/assembly/distribution.xml new file mode 100644 index 000000000..459fdb417 --- /dev/null +++ b/assembly/dss-package/src/main/assembly/distribution.xml @@ -0,0 +1,255 @@ + + + + dist + + dir + + true + dss-${dss.version} + + + + + + lib/dss-commons/ + true + true + false + true + true + + + + + + ${basedir}/../../ + . + + README* + LICENSE* + NOTICE* + + + + + + ${basedir}/../../conf + + conf + + **/* + + unix + + + + + ${basedir}/../../sbin + + sbin + + **/* + + unix + + + + + + ${basedir}/../../dss-framework/dss-framework-project-server/target/out/dss-framework-project-server/lib/ + + lib/dss-framework/dss-framework-project-server + + **/* + + + + + + + ${basedir}/../../dss-framework/dss-framework-orchestrator-server/target/out/dss-framework-orchestrator-server/lib/ + + lib/dss-framework/dss-framework-orchestrator-server/ + + **/* + + + + + + + ${basedir}/../../dss-apps/dss-apiservice-server/target/out/dss-apiservice-server/lib/ + + lib/dss-apps/dss-apiservice-server + + **/* + + + + + + + ${basedir}/../../dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/target/out/dss-workflow-server/lib/ + + lib/dss-orchestrator/dss-workflow-server/ + + **/* + + + + + + + ${basedir}/../../dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/target/out/dss-flow-execution-server/lib/ + + lib/dss-orchestrator/dss-flow-execution-server/ + + **/* + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-datachecker-appconn/target/out/ + + dss-appconns + + **/* + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-eventchecker-appconn/target/out/ + + dss-appconns + + **/* + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-orchestrator-framework-appconn/target/out/ + + dss-appconns + + **/* + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-workflow-appconn/target/out/ + + dss-appconns + + **/* + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-schedulis-appconn/target/out/ + + dss-appconns + + **/* + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-visualis-appconn/target/out/ + + dss-appconns + + **/* + + + + + + + + + + ${basedir}/../../dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/target/out/ + + dss-appconns + + **/* + + + + + + ${basedir}/../../dss-appconn/linkis-appconn-engineplugin/target/ + + dss-appconns + + *.zip + + + + + + ${basedir}/../../plugins/linkis/dss-gateway-support/target/ + + lib/dss-plugins/linkis/dss-gateway/ + + *${dss.version}.jar + + + + + + ${basedir}/../../plugins/azkaban/linkis-jobtype/target/ + + lib/dss-plugins/azkaban/linkis-jobtype/ + + *${dss.version}.zip + + + + + + + ${basedir}/../../dss-apps/dss-datapipe-server/target/out/dss-datapipe-server/lib/ + + lib/dss-apps/dss-datapipe-server + + **/* + + + + diff --git a/assembly/pom.xml b/assembly/pom.xml index 539581596..7ada30d10 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -1,9 +1,8 @@ - - lib - true - true - false - true - true - - + @@ -47,11 +34,13 @@ + + - ${project.parent.basedir}/conf/ + ${project.parent.basedir}/assembly/config/ - conf + config **/* @@ -60,7 +49,7 @@ - ${project.parent.basedir}/bin/ + ${project.parent.basedir}/assembly/bin/ bin @@ -71,7 +60,7 @@ - ${project.parent.basedir}/db/ + ${project.parent.basedir}/db db @@ -79,118 +68,16 @@ - + - ${project.parent.basedir}/dss-server/target/ + ${project.parent.basedir}/assembly/dss-package/target/out - share/dss/dss-server + - *.zip - - - - - - - - ${project.parent.basedir}/datachecker-appjoint/target/ - - share/appjoints/datachecker - - *.zip - - - - - - ${project.parent.basedir}/eventchecker-appjoint/target/ - - share/appjoints/eventchecker - - *.zip - - - - - - ${project.parent.basedir}/dss-azkaban-scheduler-appjoint/target/ - - share/appjoints/schedulis - - *.zip - - - - - - ${project.parent.basedir}/dss-flow-execution-entrance/target/ - - share/dss/dss-flow-execution-entrance - - *.zip - - - - - - - - ${project.parent.basedir}/visualis-appjoint/appjoint/target/ - - share/appjoints/visualis - - *.zip - - - - - - - ${project.parent.basedir}/qualitis-appjoint/appjoint/target/ - - share/appjoints/qualitis - - *.zip - - - - - - - ${project.parent.basedir}/visualis-appjoint/server/assembly/target/ - - share/visualis-server/ - - *.zip + **/* - - - - ${project.parent.basedir}/plugins/linkis/linkis-appjoint-entrance/target/ - - share/plugins/linkis/linkis-appjoint-entrance - - *.zip - - - - - - - - - - - - - - - - - - diff --git a/bin/checkServices.sh b/bin/checkServices.sh deleted file mode 100644 index 72df04be4..000000000 --- a/bin/checkServices.sh +++ /dev/null @@ -1,91 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -#!/bin/sh -source ~/.bash_profile - -shellDir=`dirname $0` -workDir=`cd ${shellDir}/..;pwd` - -##load config -export LINKIS_DSS_CONF_FILE=${LINKIS_DSS_CONF_FILE:-"${workDir}/conf/config.sh"} -export DISTRIBUTION=${DISTRIBUTION:-"${workDir}/conf/config.sh"} -source ${LINKIS_DSS_CONF_FILE} -source ${DISTRIBUTION} - -MICRO_SERVICE_NAME=$1 -MICRO_SERVICE_IP=$2 -MICRO_SERVICE_PORT=$3 - -local_host="`hostname --fqdn`" - -ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}'|awk 'NR==1') - -function isLocal(){ - if [ "$1" == "127.0.0.1" ];then - return 0 - elif [ $1 == "localhost" ]; then - return 0 - elif [ $1 == $local_host ]; then - return 0 - elif [ $1 == $ipaddr ]; then - return 0 - fi - return 1 -} - -function executeCMD(){ - isLocal $1 - flag=$? - echo "Is local "$flag - if [ $flag == "0" ];then - eval $2 - else - ssh -p $SSH_PORT $1 $2 - fi - -} - -#echo "<--------------------------------------------------------------------------->" -#echo "Start to Check if your microservice:$MICRO_SERVICE_NAME is normal via telnet" -#echo "" -#if ! executeCMD $SERVER_IP "test -e $DSS_INSTALL_HOME/$MICRO_SERVICE_NAME"; then -# echo "$MICRO_SERVICE_NAME is not installed,the check steps will be skipped" -# exit 0 -#fi -echo "===========================================================" -echo $MICRO_SERVICE_NAME -echo $MICRO_SERVICE_IP -echo $MICRO_SERVICE_PORT -echo "===========================================================" - -if [ $MICRO_SERVICE_NAME == "visualis-server" ] && [ $MICRO_SERVICE_IP == "127.0.0.1" ]; then - MICRO_SERVICE_IP=$ipaddr -fi - -result=`echo -e "\n" | telnet $MICRO_SERVICE_IP $MICRO_SERVICE_PORT 2>/dev/null | grep Connected | wc -l` -if [ $result -eq 1 ]; then - echo "$MICRO_SERVICE_NAME is ok." -else - echo "<--------------------------------------------------------------------------->" - echo "ERROR your $MICRO_SERVICE_NAME microservice is not start successful !!! ERROR logs as follows :" - echo "PLEAESE CHECK DETAIL LOG,LOCATION:$DSS_INSTALL_HOME/$MICRO_SERVICE_NAME/logs/linkis.out" - echo '<------------------------------------------------------------->' - executeCMD $MICRO_SERVICE_IP "tail -n 50 $DSS_INSTALL_HOME/$MICRO_SERVICE_NAME/logs/*.out" - echo '<-------------------------------------------------------------->' - echo "PLEAESE CHECK DETAIL LOG,LOCATION:$DSS_INSTALL_HOME/$MICRO_SERVICE_NAME/logs/linkis.out" - exit 1 -fi - diff --git a/bin/install.sh b/bin/install.sh deleted file mode 100644 index 170b03360..000000000 --- a/bin/install.sh +++ /dev/null @@ -1,583 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -#!/bin/sh -#Actively load user env - -source ~/.bash_profile - -shellDir=`dirname $0` -workDir=`cd ${shellDir}/..;pwd` - -#To be compatible with MacOS and Linux -txt="" -if [[ "$OSTYPE" == "darwin"* ]]; then - txt="''" -elif [[ "$OSTYPE" == "linux-gnu" ]]; then - # linux - txt="" -elif [[ "$OSTYPE" == "cygwin" ]]; then - echo "dss not support Windows operating system" - exit 1 -elif [[ "$OSTYPE" == "msys" ]]; then - echo "dss not support Windows operating system" - exit 1 -elif [[ "$OSTYPE" == "win32" ]]; then - echo "dss not support Windows operating system" - exit 1 -elif [[ "$OSTYPE" == "freebsd"* ]]; then - txt="" -else - echo "Operating system unknown, please tell us(submit issue) for better service" - exit 1 -fi - -function isSuccess(){ -if [ $? -ne 0 ]; then - echo "Failed to " + $1 - exit 1 -else - echo "Succeed to" + $1 -fi -} - -function checkJava(){ - java -version - isSuccess "execute java --version" -} - -function checkExternalServer(){ -echo "telnet check for your $SERVER_NAME, if you wait for a long time,may be your $SERVER_NAME does not prepared" -result=`echo -e "\n" | telnet $EXTERNAL_SERVER_IP $EXTERNAL_SERVER_PORT 2>/dev/null | grep Connected | wc -l` -if [ $result -eq 1 ]; then - echo "$SERVER_NAME is OK." -else - echo "$SERVER_NAME is Bad. You need to prepare the' $SERVER_NAME ' environment in advance" - exit 1 -fi -} - - -say() { - printf 'check command fail \n %s\n' "$1" -} - -err() { - say "$1" >&2 - exit 1 -} - -check_cmd() { - command -v "$1" > /dev/null 2>&1 -} - -need_cmd() { - if ! check_cmd "$1"; then - err "need '$1' (command not found)" - fi -} - -#check env -sh ${workDir}/bin/checkEnv.sh -isSuccess "check env" - -##load config -echo "step1:load config" -export DSS_CONFIG_PATH=${DSS_CONFIG_PATH:-"${workDir}/conf/config.sh"} -export DSS_DB_CONFIG_PATH=${DSS_DB_CONFIG_PATH:-"${workDir}/conf/db.sh"} -export DISTRIBUTION=${DISTRIBUTION:-"${workDir}/conf/config.sh"} -source ${DSS_CONFIG_PATH} -source ${DSS_DB_CONFIG_PATH} -source ${DISTRIBUTION} -isSuccess "load config" - -local_host="`hostname --fqdn`" -ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}'|awk 'NR==1') - -function isLocal(){ - if [ "$1" == "127.0.0.1" ];then - return 0 - elif [ $1 == "localhost" ]; then - return 0 - elif [ $1 == $local_host ]; then - return 0 - elif [ $1 == $ipaddr ]; then - return 0 - fi - return 1 -} - -function executeCMD(){ - isLocal $1 - flag=$? - if [ $flag == "0" ];then - echo "Is local execution:$2" - eval $2 - else - echo "Is remote execution:$2" - ssh -p $SSH_PORT $1 $2 - fi -} - -function copyFile(){ - isLocal $1 - flag=$? - src=$2 - dest=$3 - if [ $flag == "0" ];then - echo "Is local cp " - eval "cp -r $src $dest" - else - echo "Is remote cp " - scp -r -P $SSH_PORT $src $1:$dest - fi -} - -##install mode choice -if [ "$INSTALL_MODE" == "" ];then - echo "Please enter the mode selection such as: 1" - echo " 1: Lite" - echo " 2: Simple" - echo " 3: Standard" - echo "" - read -p "Please input the choice:" idx - INSTALL_MODE=$idx -fi - -if [[ '1' = "$INSTALL_MODE" ]];then - echo "You chose lite installation mode" - checkJava - SERVER_NAME=MYSQL - EXTERNAL_SERVER_IP=$MYSQL_HOST - EXTERNAL_SERVER_PORT=$MYSQL_PORT - checkExternalServer -elif [[ '2' = "$INSTALL_MODE" ]];then - echo "You chose sample installation mode" - checkJava - SERVER_NAME=MYSQL - EXTERNAL_SERVER_IP=$MYSQL_HOST - EXTERNAL_SERVER_PORT=$MYSQL_PORT - checkExternalServer -elif [[ '3' = "$INSTALL_MODE" ]];then - echo "You chose Standard installation mode" - #check for Java - checkJava - #check for mysql - SERVER_NAME=MYSQL - EXTERNAL_SERVER_IP=$MYSQL_HOST - EXTERNAL_SERVER_PORT=$MYSQL_PORT - checkExternalServer - #check qualitis serivice - SERVER_NAME=Qualitis - EXTERNAL_SERVER_IP=$QUALITIS_ADRESS_IP - EXTERNAL_SERVER_PORT=$QUALITIS_ADRESS_PORT - if [[ $IGNORECHECK = "" ]];then - checkExternalServer - fi - #check azkaban serivice - SERVER_NAME=AZKABAN - EXTERNAL_SERVER_IP=$AZKABAN_ADRESS_IP - EXTERNAL_SERVER_PORT=$AZKABAN_ADRESS_PORT - if [[ $IGNORECHECK = "" ]];then - checkExternalServer - fi -else - echo "no choice,exit!" - exit 1 -fi - -##init db -echo "Do you want to clear DSS table information in the database?" -echo " 1: Do not execute table-building statements" -echo " 2: Dangerous! Clear all data and rebuild the tables." -echo "" - -MYSQL_INSTALL_MODE=1 - -read -p "Please input the choice:" idx -if [[ '2' = "$idx" ]];then - MYSQL_INSTALL_MODE=2 - echo "You chose Rebuild the table" -elif [[ '1' = "$idx" ]];then - MYSQL_INSTALL_MODE=1 - echo "You chose not execute table-building statements" -else - echo "no choice,exit!" - exit 1 -fi - -echo "create hdfs directory and local directory" -if [ "$WORKSPACE_USER_ROOT_PATH" != "" ] -then - localRootDir=$WORKSPACE_USER_ROOT_PATH - if [[ $WORKSPACE_USER_ROOT_PATH == file://* ]];then - localRootDir=${WORKSPACE_USER_ROOT_PATH#file://} - mkdir -p $localRootDir/$deployUser - sudo chmod -R 775 $localRootDir/$deployUser - elif [[ $WORKSPACE_USER_ROOT_PATH == hdfs://* ]];then - localRootDir=${WORKSPACE_USER_ROOT_PATH#hdfs://} - hdfs dfs -mkdir -p $localRootDir/$deployUser - hdfs dfs -chmod -R 775 $localRootDir/$deployUser - else - echo "does not support $WORKSPACE_USER_ROOT_PATH filesystem types" - fi -isSuccess "create $WORKSPACE_USER_ROOT_PATH directory" -fi - - -if [ "$RESULT_SET_ROOT_PATH" != "" ] -then - localRootDir=$RESULT_SET_ROOT_PATH - if [[ $RESULT_SET_ROOT_PATH == file://* ]];then - localRootDir=${RESULT_SET_ROOT_PATH#file://} - mkdir -p $localRootDir/$deployUser - sudo chmod -R 775 $localRootDir/$deployUser - elif [[ $RESULT_SET_ROOT_PATH == hdfs://* ]];then - localRootDir=${RESULT_SET_ROOT_PATH#hdfs://} - hdfs dfs -mkdir -p $localRootDir/$deployUser - hdfs dfs -chmod -R 775 $localRootDir/$deployUser - else - echo "does not support $RESULT_SET_ROOT_PATH filesystem types" - fi -isSuccess "create $RESULT_SET_ROOT_PATH directory" -fi - - -if [ "$WDS_SCHEDULER_PATH" != "" ] -then - localRootDir=$WDS_SCHEDULER_PATH - if [[ $WDS_SCHEDULER_PATH == file://* ]];then - localRootDir=${WDS_SCHEDULER_PATH#file://} - mkdir -p $localRootDir - sudo chmod -R 775 $localRootDir - elif [[ $WDS_SCHEDULER_PATH == hdfs://* ]];then - localRootDir=${WDS_SCHEDULER_PATH#hdfs://} - hdfs dfs -mkdir -p $localRootDir - hdfs dfs -chmod -R 775 $localRootDir - else - echo "does not support $WDS_SCHEDULER_PATH filesystem types" - fi -isSuccess "create $WDS_SCHEDULER_PATH directory" -fi - - -##init db -if [[ '2' = "$MYSQL_INSTALL_MODE" ]];then - mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source ${workDir}/db/dss_ddl.sql" - isSuccess "source dss_ddl.sql" - LOCAL_IP=$ipaddr - if [ $GATEWAY_INSTALL_IP == "127.0.0.1" ];then - echo "GATEWAY_INSTALL_IP is equals 127.0.0.1 ,we will change it to ip address" - GATEWAY_INSTALL_IP_2=$LOCAL_IP - else - GATEWAY_INSTALL_IP_2=$GATEWAY_INSTALL_IP - fi - #echo $GATEWAY_INSTALL_IP_2 - sed -i "s/GATEWAY_INSTALL_IP_2/$GATEWAY_INSTALL_IP_2/g" ${workDir}/db/dss_dml.sql - sed -i "s/GATEWAY_PORT/$GATEWAY_PORT/g" ${workDir}/db/dss_dml.sql - mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source ${workDir}/db/dss_dml.sql" - isSuccess "source dss_dml.sql" - - if [[ '2' = "$INSTALL_MODE" ]] || [[ '3' = "$INSTALL_MODE" ]];then - echo "visualis support,visualis database will be initialized !" - if [ $VISUALIS_NGINX_IP == "127.0.0.1" ]||[ $VISUALIS_NGINX_IP == "0.0.0.0" ];then - echo "VISUALIS_NGINX_IP is equals $VISUALIS_NGINX_IP ,we will change it to ip address" - VISUALIS_NGINX_IP_2=$LOCAL_IP - else - VISUALIS_NGINX_IP_2=$VISUALIS_NGINX_IP - fi - #echo $VISUALIS_NGINX_IP_2 - sed -i "s/VISUALIS_NGINX_IP_2/$VISUALIS_NGINX_IP_2/g" ${workDir}/db/visualis.sql - sed -i "s/VISUALIS_NGINX_PORT/$VISUALIS_NGINX_PORT/g" ${workDir}/db/visualis.sql - mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source ${workDir}/db/visualis.sql" - isSuccess "source visualis.sql" - mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source ${workDir}/db/davinci.sql" - isSuccess "source davinci.sql" - fi - - if [[ '3' = "$INSTALL_MODE" ]];then - echo "azkaban and qualitis support, azkaban and qualitis database will be initialized !" - #azkaban - if [ $AZKABAN_ADRESS_IP == "127.0.0.1" ];then - echo "AZKABAN_ADRESS_IP is equals 127.0.0.1 ,we will change it to ip address" - AZKABAN_ADRESS_IP_2=$LOCAL_IP - else - AZKABAN_ADRESS_IP_2=$AZKABAN_ADRESS_IP - fi - echo $AZKABAN_ADRESS_IP_2 - sed -i "s/AZKABAN_ADRESS_IP_2/$AZKABAN_ADRESS_IP_2/g" ${workDir}/db/azkaban.sql - sed -i "s/AZKABAN_ADRESS_PORT/$AZKABAN_ADRESS_PORT/g" ${workDir}/db/azkaban.sql - mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source ${workDir}/db/azkaban.sql" - isSuccess "source azkaban.sql" - #qualitis - if [ $QUALITIS_ADRESS_IP == "127.0.0.1" ];then - echo "QUALITIS_ADRESS_IP is equals 127.0.0.1 ,we will change it to ip address" - QUALITIS_ADRESS_IP_2=$LOCAL_IP - else - QUALITIS_ADRESS_IP_2=$QUALITIS_ADRESS_IP - fi - echo $QUALITIS_ADRESS_IP_2 - sed -i "s/QUALITIS_ADRESS_IP_2/$QUALITIS_ADRESS_IP_2/g" ${workDir}/db/qualitis.sql - sed -i "s/QUALITIS_ADRESS_PORT/$QUALITIS_ADRESS_PORT/g" ${workDir}/db/qualitis.sql - mysql -h$MYSQL_HOST -P$MYSQL_PORT -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DB --default-character-set=utf8 -e "source ${workDir}/db/qualitis.sql" - isSuccess "source qualitis.sql" - fi -fi - -##Deal special symbol '#' -HIVE_META_PASSWORD=$(echo ${HIVE_META_PASSWORD//'#'/'\#'}) -MYSQL_PASSWORD=$(echo ${MYSQL_PASSWORD//'#'/'\#'}) - -###linkis Eurkea info -SERVER_IP=$EUREKA_INSTALL_IP -SERVER_PORT=$EUREKA_PORT -SERVER_HOME=$DSS_INSTALL_HOME - -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi -EUREKA_URL=http://$SERVER_IP:$EUREKA_PORT/eureka/ - -##function start -function installPackage(){ -echo "start to install $SERVERNAME" -echo "$SERVERNAME-step1: create dir" -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi - -if ! executeCMD $SERVER_IP "test -e $SERVER_HOME"; then - executeCMD $SERVER_IP "sudo mkdir -p $SERVER_HOME;sudo chown -R $deployUser:$deployUser $SERVER_HOME" - isSuccess "create the dir of $SERVERNAME" -fi - -echo "$SERVERNAME-step2:copy install package" -copyFile $SERVER_IP ${workDir}/share/$PACKAGE_DIR/$SERVERNAME.zip $SERVER_HOME - -if ! executeCMD $SERVER_IP "test -e $SERVER_HOME/lib"; then - copyFile $SERVER_IP ${workDir}/lib $SERVER_HOME -fi - -#copyFile $SERVER_IP ${workDir}/lib $SERVER_HOME -isSuccess "copy ${SERVERNAME}.zip" -executeCMD $SERVER_IP "cd $SERVER_HOME/;rm -rf $SERVERNAME-bak; mv -f $SERVERNAME $SERVERNAME-bak" -executeCMD $SERVER_IP "cd $SERVER_HOME/;unzip $SERVERNAME.zip > /dev/null" -executeCMD $SERVER_IP "cd $SERVER_HOME/;scp -r lib/* $SERVER_HOME/$SERVERNAME/lib" -isSuccess "unzip ${SERVERNAME}.zip" - -echo "$SERVERNAME-step3:subsitution conf" -SERVER_CONF_PATH=$SERVER_HOME/$SERVERNAME/conf/application.yml -executeCMD $SERVER_IP "sed -i \"s#port:.*#port: $SERVER_PORT#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#defaultZone:.*#defaultZone: $EUREKA_URL#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#hostname:.*#hostname: $SERVER_IP#g\" $SERVER_CONF_PATH" -isSuccess "subsitution conf of $SERVERNAME" -} -##function end - -##function start -function installVisualis(){ -echo "start to install $SERVERNAME" -echo "$SERVERNAME-step1: create dir" -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi - -if ! executeCMD $SERVER_IP "test -e $SERVER_HOME"; then - executeCMD $SERVER_IP "sudo mkdir -p $SERVER_HOME;sudo chown -R $deployUser:$deployUser $SERVER_HOME" - isSuccess "create the dir of $SERVERNAME" -fi - -echo "$SERVERNAME-step2:copy install package" -copyFile $SERVER_IP ${workDir}/share/$PACKAGE_DIR/$SERVERNAME.zip $SERVER_HOME -isSuccess "copy ${SERVERNAME}.zip" -executeCMD $SERVER_IP "cd $SERVER_HOME/;rm -rf $SERVERNAME-bak; mv -f $SERVERNAME $SERVERNAME-bak" -executeCMD $SERVER_IP "cd $SERVER_HOME/;unzip $SERVERNAME.zip > /dev/null" -isSuccess "unzip ${SERVERNAME}.zip" -} -##function end - - -##function start -function installAppjoints(){ -echo "start to install $APPJOINTNAME" -echo "$APPJOINTNAME Install-step1: create dir" -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi - -if ! executeCMD $SERVER_IP "test -e $SERVER_HOME/$APPJOINTPARENT"; then - executeCMD $SERVER_IP "sudo mkdir -p $SERVER_HOME/$APPJOINTPARENT;sudo chown -R $deployUser:$deployUser $SERVER_HOME/$APPJOINTPARENT" - isSuccess "create the dir of $SERVER_HOME/$APPJOINTPARENT;" -fi - -echo "$APPJOINTNAME-step2:copy install package" -copyFile $SERVER_IP $workDir/share/appjoints/$APPJOINTNAME/*.zip $SERVER_HOME/$APPJOINTPARENT -isSuccess "copy ${APPJOINTNAME}.zip" -executeCMD $SERVER_IP "cd $SERVER_HOME/$APPJOINTPARENT/;unzip -o dss-*-appjoint.zip > /dev/null;rm -rf dss-*-appjoint.zip" -isSuccess "install ${APPJOINTNAME}.zip" -} -##function end - -##dss-Server install -PACKAGE_DIR=dss/dss-server -SERVERNAME=dss-server -SERVER_IP=$DSS_SERVER_INSTALL_IP -SERVER_PORT=$DSS_SERVER_PORT -SERVER_HOME=$DSS_INSTALL_HOME -###install Dss-Server -installPackage -###update Dss-Server linkis.properties -echo "$SERVERNAME-step4:update linkis.properties" -SERVER_CONF_PATH=$SERVER_HOME/$SERVERNAME/conf/linkis.properties -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.server.mybatis.datasource.url.*#wds.linkis.server.mybatis.datasource.url=jdbc:mysql://${MYSQL_HOST}:${MYSQL_PORT}/${MYSQL_DB}?characterEncoding=UTF-8#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.server.mybatis.datasource.username.*#wds.linkis.server.mybatis.datasource.username=$MYSQL_USER#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.server.mybatis.datasource.password.*#***REMOVED***$MYSQL_PASSWORD#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.dss.appjoint.scheduler.azkaban.address.*#wds.dss.appjoint.scheduler.azkaban.address=http://${AZKABAN_ADRESS_IP}:${AZKABAN_ADRESS_PORT}#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.gateway.ip.*#wds.linkis.gateway.ip=$GATEWAY_INSTALL_IP#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.gateway.port.*#wds.linkis.gateway.port=$GATEWAY_PORT#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.dss.appjoint.scheduler.project.store.dir.*#wds.dss.appjoint.scheduler.project.store.dir=$WDS_SCHEDULER_PATH#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "echo "$deployUser=$deployUser" >> $SERVER_HOME/$SERVERNAME/conf/token.properties" -isSuccess "subsitution linkis.properties of $SERVERNAME" -echo "<----------------$SERVERNAME:end------------------->" -echo "" - -if [[ '2' = "$INSTALL_MODE" ]]||[[ '3' = "$INSTALL_MODE" ]];then -##Flow execution Install -PACKAGE_DIR=dss/dss-flow-execution-entrance -SERVERNAME=dss-flow-execution-entrance -SERVER_IP=$FLOW_EXECUTION_INSTALL_IP -SERVER_PORT=$FLOW_EXECUTION_PORT -SERVER_HOME=$DSS_INSTALL_HOME -###Install flow execution -installPackage -###Update flow execution linkis.properties -echo "$SERVERNAME-step4:update linkis.properties" -SERVER_CONF_PATH=$SERVER_HOME/$SERVERNAME/conf/linkis.properties -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.entrance.config.logPath.*#wds.linkis.entrance.config.logPath=$WORKSPACE_USER_ROOT_PATH#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.resultSet.store.path.*#wds.linkis.resultSet.store.path=$RESULT_SET_ROOT_PATH#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.gateway.url.*#wds.linkis.gateway.url=http://${GATEWAY_INSTALL_IP}:${GATEWAY_PORT}#g\" $SERVER_CONF_PATH" -isSuccess "subsitution linkis.properties of $SERVERNAME" -echo "<----------------$SERVERNAME:end------------------->" -echo "" -##Appjoint entrance Install -PACKAGE_DIR=plugins/linkis/linkis-appjoint-entrance -SERVERNAME=linkis-appjoint-entrance -SERVER_IP=$APPJOINT_ENTRANCE_INSTALL_IP -SERVER_PORT=$APPJOINT_ENTRANCE_PORT -SERVER_HOME=$DSS_INSTALL_HOME -###Install appjoint entrance -installPackage -###Update appjoint entrance linkis.properties -echo "$SERVERNAME-step4:update linkis.properties" -SERVER_CONF_PATH=$SERVER_HOME/$SERVERNAME/conf/linkis.properties -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.entrance.config.logPath.*#wds.linkis.entrance.config.logPath=$WORKSPACE_USER_ROOT_PATH#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.resultSet.store.path.*#wds.linkis.resultSet.store.path=$RESULT_SET_ROOT_PATH#g\" $SERVER_CONF_PATH" -isSuccess "subsitution linkis.properties of $SERVERNAME" -echo "<----------------$SERVERNAME:end------------------->" -echo "" -##visualis-server Install -PACKAGE_DIR=visualis-server -SERVERNAME=visualis-server -SERVER_IP=$VISUALIS_SERVER_INSTALL_IP -SERVER_PORT=$VISUALIS_SERVER_PORT -SERVER_HOME=$DSS_INSTALL_HOME -###install visualis-server -installVisualis -###update visualis-server linkis.properties -echo "$SERVERNAME-step4:update linkis.properties" -SERVER_CONF_PATH=$SERVER_HOME/$SERVERNAME/conf/linkis.properties -if [ $VISUALIS_NGINX_IP == "127.0.0.1" ]||[ $VISUALIS_NGINX_IP == "0.0.0.0" ]; then - VISUALIS_NGINX_IP=$ipaddr -fi -if [ $VISUALIS_SERVER_INSTALL_IP == "127.0.0.1" ]||[ $VISUALIS_SERVER_INSTALL_IP == "0.0.0.0" ]; then - VISUALIS_SERVER_INSTALL_IP=$ipaddr -fi -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.entrance.config.logPath.*#wds.linkis.entrance.config.logPath=$WORKSPACE_USER_ROOT_PATH#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.linkis.resultSet.store.path.*#wds.linkis.resultSet.store.path=$RESULT_SET_ROOT_PATH#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.dss.visualis.gateway.ip.*#wds.dss.visualis.gateway.ip=$GATEWAY_INSTALL_IP#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#wds.dss.visualis.gateway.port.*#wds.dss.visualis.gateway.port=$GATEWAY_PORT#g\" $SERVER_CONF_PATH" -SERVER_CONF_PATH=$SERVER_HOME/$SERVERNAME/conf/application.yml -executeCMD $SERVER_IP "sed -i \"s#address: 127.0.0.1#address: $VISUALIS_SERVER_INSTALL_IP#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#port: 9007#port: $VISUALIS_SERVER_PORT#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#url: http://0.0.0.0:0000/dss/visualis#url: http://$VISUALIS_NGINX_IP:$VISUALIS_NGINX_PORT/dss/visualis#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#address: 0.0.0.0#address: $VISUALIS_NGINX_IP#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#port: 0000#port: $VISUALIS_NGINX_PORT#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#defaultZone: http://127.0.0.1:20303/eureka/#defaultZone: http://$EUREKA_INSTALL_IP:$EUREKA_PORT/eureka/#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#url: jdbc:mysql://127.0.0.1:3306/xxx?characterEncoding=UTF-8#url: jdbc:mysql://$MYSQL_HOST:$MYSQL_PORT/$MYSQL_DB?characterEncoding=UTF-8#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#username: xxx#username: $MYSQL_USER#g\" $SERVER_CONF_PATH" -executeCMD $SERVER_IP "sed -i \"s#password: xxx#password: $MYSQL_PASSWORD#g\" $SERVER_CONF_PATH" -isSuccess "subsitution linkis.properties of $SERVERNAME" -echo "<----------------$SERVERNAME:end------------------->" -echo "" -#APPJOINTS INSTALL -echo "<----------------datachecker appjoint install start------------------->" -APPJOINTPARENT=dss-appjoints -APPJOINTNAME=datachecker -#datachecker appjoint install -installAppjoints -echo "$APPJOINTNAME:subsitution conf" -APPJOINTNAME_CONF_PATH_PATENT=$SERVER_HOME/$APPJOINTPARENT/$APPJOINTNAME/appjoint.properties -executeCMD $SERVER_IP "sed -i \"s#job.datachecker.jdo.option.url.*#job.datachecker.jdo.option.url=$HIVE_META_URL#g\" $APPJOINTNAME_CONF_PATH_PATENT" -executeCMD $SERVER_IP "sed -i \"s#job.datachecker.jdo.option.username.*#job.datachecker.jdo.option.username=$HIVE_META_USER#g\" $APPJOINTNAME_CONF_PATH_PATENT" -executeCMD $SERVER_IP "sed -i \"s#job.datachecker.jdo.option.password.*#job.datachecker.jdo.option.password=$HIVE_META_PASSWORD#g\" $APPJOINTNAME_CONF_PATH_PATENT" -isSuccess "subsitution conf of datachecker" -echo "<----------------datachecker appjoint install end------------------->" -echo "" -echo "<----------------eventchecker appjoint install start------------------->" -APPJOINTPARENT=dss-appjoints -APPJOINTNAME=eventchecker -#eventchecker appjoint install -installAppjoints -echo "$APPJOINTNAME:subsitution conf" -APPJOINTNAME_CONF_PATH_PATENT=$SERVER_HOME/$APPJOINTPARENT/$APPJOINTNAME/appjoint.properties -executeCMD $SERVER_IP "sed -i \"s#msg.eventchecker.jdo.option.url.*#msg.eventchecker.jdo.option.url=jdbc:mysql://${MYSQL_HOST}:${MYSQL_PORT}/${MYSQL_DB}?characterEncoding=UTF-8#g\" $APPJOINTNAME_CONF_PATH_PATENT" -executeCMD $SERVER_IP "sed -i \"s#msg.eventchecker.jdo.option.username.*#msg.eventchecker.jdo.option.username=$MYSQL_USER#g\" $APPJOINTNAME_CONF_PATH_PATENT" -executeCMD $SERVER_IP "sed -i \"s#msg.eventchecker.jdo.option.password.*#msg.eventchecker.jdo.option.password=$MYSQL_PASSWORD#g\" $APPJOINTNAME_CONF_PATH_PATENT" -isSuccess "subsitution conf of eventchecker" -echo "<----------------$APPJOINTNAME:end------------------->" -echo "" -echo "<----------------visualis appjoint install start------------------->" -APPJOINTPARENT=dss-appjoints -APPJOINTNAME=visualis -#visualis appjoint install -installAppjoints -echo "<----------------$APPJOINTNAME:end------------------->" -fi - -##lite and sample version does not install qualitis APPJoint and scheduis APPJoint -if [[ '3' = "$INSTALL_MODE" ]];then -echo "" -echo "<----------------qualitis appjoint install start------------------->" -APPJOINTPARENT=dss-appjoints -APPJOINTNAME=qualitis -#qualitis appjoint install -installAppjoints -APPJOINTNAME_CONF_PATH_PATENT=$SERVER_HOME/$APPJOINTPARENT/$APPJOINTNAME/appjoint.properties -executeCMD $SERVER_IP "sed -i \"s#baseUrl=http://127.0.0.1:8090#baseUrl=http://$QUALITIS_ADRESS_IP:$QUALITIS_ADRESS_PORT#g\" $APPJOINTNAME_CONF_PATH_PATENT" -isSuccess "subsitution conf of qualitis" -echo "<----------------$APPJOINTNAME:end------------------->" -echo "" -echo "<----------------schedulis appjoint install start------------------->" -APPJOINTPARENT=dss-appjoints -APPJOINTNAME=schedulis -#schedulis appjoint install -installAppjoints -isSuccess "subsitution conf of schedulis" -echo "<----------------$APPJOINTNAME:end------------------->" -fi diff --git a/bin/start-all.sh b/bin/start-all.sh deleted file mode 100644 index 98a07f6bd..000000000 --- a/bin/start-all.sh +++ /dev/null @@ -1,189 +0,0 @@ -#!/usr/bin/env bash -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# Start all dss applications -info="We will start all dss applications, it will take some time, please wait" -echo ${info} - -#Actively load user env -source /etc/profile -source ~/.bash_profile - -shellDir=`dirname $0` - -workDir=`cd ${shellDir}/..;pwd` - -CONF_DIR="${workDir}"/conf - -export LINKIS_DSS_CONF_FILE=${LINKIS_DSS_CONF_FILE:-"${CONF_DIR}/config.sh"} -export DISTRIBUTION=${DISTRIBUTION:-"${CONF_DIR}/config.sh"} -source $LINKIS_DSS_CONF_FILE -source ${DISTRIBUTION} -function isSuccess(){ -if [ $? -ne 0 ]; then - echo "ERROR: " + $1 - exit 1 -else - echo "INFO:" + $1 -fi -} -local_host="`hostname --fqdn`" - -ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}'|awk 'NR==1') - -function isLocal(){ - if [ "$1" == "127.0.0.1" ];then - return 0 - elif [ $1 == "localhost" ]; then - return 0 - elif [ $1 == $local_host ]; then - return 0 - elif [ $1 == $ipaddr ]; then - return 0 - fi - return 1 -} - -function executeCMD(){ - isLocal $1 - flag=$? - echo "Is local "$flag - if [ $flag == "0" ];then - eval $2 - else - ssh -p $SSH_PORT $1 $2 - fi - -} - -#if there is no LINKIS_INSTALL_HOME，we need to source config again -if [ -z ${DSS_INSTALL_HOME} ];then - echo "Warning: DSS_INSTALL_HOME does not exist, we will source config" - if [ ! -f "${LINKIS_DSS_CONF_FILE}" ];then - echo "Error: can not find config file, start applications failed" - exit 1 - else - source ${LINKIS_DSS_CONF_FILE} - fi -fi - -function startApp(){ -echo "<-------------------------------->" -echo "Begin to start $SERVER_NAME" -SERVER_BIN=${DSS_INSTALL_HOME}/${SERVER_NAME}/bin -#echo $SERVER_BIN -SERVER_LOCAL_START_CMD="dos2unix ${SERVER_BIN}/* > /dev/null 2>&1; dos2unix ${SERVER_BIN}/../conf/* > /dev/null 2>&1;sh ${SERVER_BIN}/start-${SERVER_NAME}.sh > /dev/null 2>&1 &" -SERVER_REMOTE_START_CMD="source /etc/profile;source ~/.bash_profile;cd ${SERVER_BIN}; dos2unix ./* > /dev/null 2>&1; dos2unix ../conf/* > /dev/null 2>&1; sh start-${SERVER_NAME}.sh > /dev/null 2>&1" - -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi - -if ! executeCMD $SERVER_IP "test -e $SERVER_BIN"; then - echo "<-------------------------------->" - echo "$SERVER_NAME is not installed,the start steps will be skipped" - echo "<-------------------------------->" - return -fi - -isLocal $SERVER_IP -flag=$? -echo "Is local "$flag -if [ $flag == "0" ];then - eval $SERVER_LOCAL_START_CMD -else - ssh -p $SSH_PORT $SERVER_IP $SERVER_REMOTE_START_CMD -fi -isSuccess "End to start $SERVER_NAME" -echo "<-------------------------------->" -sleep 15 #for Eureka register -} - -#dss-server -SERVER_NAME=dss-server -SERVER_IP=$DSS_SERVER_INSTALL_IP -startApp - -#dss-flow-execution-entrance -SERVER_NAME=dss-flow-execution-entrance -SERVER_IP=$FLOW_EXECUTION_INSTALL_IP -startApp - -#dss-flow-execution-entrance -SERVER_NAME=linkis-appjoint-entrance -SERVER_IP=$APPJOINT_ENTRANCE_INSTALL_IP -startApp - -#visualis-server -SERVER_NAME=visualis-server -SERVER_IP=$VISUALIS_SERVER_INSTALL_IP -startApp - -echo "" -echo "Start to check all dss microservice" -echo "" - -function checkServer(){ -echo "<-------------------------------->" -echo "Begin to check $SERVER_NAME" -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi - -SERVER_BIN=${SERVER_HOME}/${SERVER_NAME}/bin - -if ! executeCMD $SERVER_IP "test -e ${DSS_INSTALL_HOME}/${SERVER_NAME}"; then - echo "$SERVER_NAME is not installed,the checkServer steps will be skipped" - return -fi - -sh $workDir/bin/checkServices.sh $SERVER_NAME $SERVER_IP $SERVER_PORT -isSuccess "start $SERVER_NAME " -sleep 3 -echo "<-------------------------------->" -} - -#check dss-server -SERVER_NAME=dss-server -SERVER_IP=$DSS_SERVER_INSTALL_IP -SERVER_PORT=$DSS_SERVER_PORT -checkServer - - -#check dss-flow-execution-entrance -SERVER_NAME=dss-flow-execution-entrance -SERVER_IP=$FLOW_EXECUTION_INSTALL_IP -SERVER_PORT=$FLOW_EXECUTION_PORT -checkServer - -#check linkis-appjoint-entrance -SERVER_NAME=linkis-appjoint-entrance -SERVER_IP=$APPJOINT_ENTRANCE_INSTALL_IP -SERVER_PORT=$APPJOINT_ENTRANCE_PORT -checkServer - - -#check visualis-server -sleep 10 #visualis service need more time to register -SERVER_NAME=visualis-server -SERVER_IP=$VISUALIS_SERVER_INSTALL_IP -SERVER_PORT=$VISUALIS_SERVER_PORT -checkServer - -echo "DSS started successfully" diff --git a/bin/stop-all.sh b/bin/stop-all.sh deleted file mode 100644 index 838b9babc..000000000 --- a/bin/stop-all.sh +++ /dev/null @@ -1,133 +0,0 @@ -#!/usr/bin/env bash -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - - - -# Start all dss applications -info="We will stop all dss applications, it will take some time, please wait" -echo ${info} - -#Actively load user env -source ~/.bash_profile - -workDir=`dirname "${BASH_SOURCE-$0}"` -workDir=`cd "$workDir"; pwd` - - -CONF_DIR="${workDir}"/../conf -export LINKIS_DSS_CONF_FILE=${LINKIS_DSS_CONF_FILE:-"${CONF_DIR}/config.sh"} -export DISTRIBUTION=${DISTRIBUTION:-"${CONF_DIR}/config.sh"} -source ${DISTRIBUTION} - -local_host="`hostname --fqdn`" -ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}'|awk 'NR==1') - -function isSuccess(){ -if [ $? -ne 0 ]; then - echo "ERROR: " + $1 - exit 1 -else - echo "INFO:" + $1 -fi -} - -function isLocal(){ - if [ "$1" == "127.0.0.1" ];then - return 0 - elif [ $1 == "localhost" ]; then - return 0 - elif [ $1 == $local_host ]; then - return 0 - elif [ $1 == $ipaddr ]; then - return 0 - fi - return 1 -} - -function executeCMD(){ - isLocal $1 - flag=$? - echo "Is local "$flag - if [ $flag == "0" ];then - eval $2 - else - ssh -p $SSH_PORT $1 $2 - fi - -} - - -#if there is no LINKIS_INSTALL_HOME，we need to source config again -if [ -z ${DSS_INSTALL_HOME} ];then - echo "Warning: DSS_INSTALL_HOME does not exist, we will source config" - if [ ! -f "${LINKIS_DSS_CONF_FILE}" ];then - echo "Error: can not find config file, stop applications failed" - exit 1 - else - source ${LINKIS_DSS_CONF_FILE} - fi -fi - -function stopAPP(){ -echo "<-------------------------------->" -echo "Begin to stop $SERVER_NAME" -SERVER_BIN=${DSS_INSTALL_HOME}/${SERVER_NAME}/bin -SERVER_LOCAL_STOP_CMD="sh ${SERVER_BIN}/stop-${SERVER_NAME}.sh" -SERVER_REMOTE_STOP_CMD="source /etc/profile;source ~/.bash_profile;cd ${SERVER_BIN}; sh stop-${SERVER_NAME}.sh " -if test -z "$SERVER_IP" -then - SERVER_IP=$local_host -fi - -if ! executeCMD $SERVER_IP "test -e ${DSS_INSTALL_HOME}/${SERVER_NAME}"; then - echo "$SERVER_NAME is not installed,the stop steps will be skipped" - return -fi - -isLocal $SERVER_IP -flag=$? -echo "Is local "$flag -if [ $flag == "0" ];then - eval $SERVER_LOCAL_STOP_CMD -else - ssh -p $SSH_PORT $SERVER_IP $SERVER_REMOTE_STOP_CMD -fi -echo "<-------------------------------->" -sleep 3 -} - -#dss-server -SERVER_NAME=dss-server -SERVER_IP=$DSS_SERVER_INSTALL_IP -stopAPP - -#dss-flow-execution-entrance -SERVER_NAME=dss-flow-execution-entrance -SERVER_IP=$FLOW_EXECUTION_INSTALL_IP -stopAPP - -#dss-flow-execution-entrance -SERVER_NAME=linkis-appjoint-entrance -SERVER_IP=$APPJOINT_ENTRANCE_INSTALL_IP -stopAPP - -#visualis-server -SERVER_NAME=visualis-server -SERVER_IP=$VISUALIS_SERVER_INSTALL_IP -stopAPP - -echo "stop-all shell script executed completely" diff --git a/conf/application-dss.yml b/conf/application-dss.yml new file mode 100644 index 000000000..b0473838d --- /dev/null +++ b/conf/application-dss.yml @@ -0,0 +1,23 @@ + +eureka: + client: + serviceUrl: + defaultZone: http://127.0.0.1:20303/eureka/ + #instance: + #prefer-ip-address: true + #instance-id: ${spring.cloud.client.ip-address}:${server.port} + #metadata-map: + #test: wedatasphere + +management: + endpoints: + web: + exposure: + include: refresh,info +logging: + config: classpath:log4j2.xml + +#mybatis: +# configuration: +# log-impl: org.apache.ibatis.logging.stdout.StdOutImpl + diff --git a/conf/config.sh b/conf/config.sh deleted file mode 100644 index 74c913ddc..000000000 --- a/conf/config.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/bin/sh - -shellDir=`dirname $0` -workDir=`cd ${shellDir}/..;pwd` - -### deploy user -deployUser=hadoop - -### The install home path of DSS，Must provided -DSS_INSTALL_HOME=$workDir - -### Specifies the user workspace, which is used to store the user's script files and log files. -### Generally local directory -WORKSPACE_USER_ROOT_PATH=file:///tmp/linkis/ -### Path to store job ResultSet：file or hdfs path -RESULT_SET_ROOT_PATH=hdfs:///tmp/linkis - -################### The install Configuration of all Micro-Services ##################### -# -# NOTICE: -# 1. If you just wanna try, the following micro-service configuration can be set without any settings. -# These services will be installed by default on this machine. -# 2. In order to get the most complete enterprise-level features, we strongly recommend that you install -# the following microservice parameters -# - -### DSS_SERVER -### This service is used to provide dss-server capability. -DSS_SERVER_INSTALL_IP=127.0.0.1 -DSS_SERVER_PORT=9004 - -### Appjoint-Entrance -### This service is used to provide Appjoint-Entrance capability. -APPJOINT_ENTRANCE_INSTALL_IP=127.0.0.1 -APPJOINT_ENTRANCE_PORT=9005 - -### Flow-Execution-Entrance -### This service is used to provide flow execution capability. -FLOW_EXECUTION_INSTALL_IP=127.0.0.1 -FLOW_EXECUTION_PORT=9006 - -### Linkis EUREKA information. -EUREKA_INSTALL_IP=127.0.0.1 # Microservices Service Registration Discovery Center -EUREKA_PORT=20303 - -### Linkis Gateway information -GATEWAY_INSTALL_IP=127.0.0.1 -GATEWAY_PORT=9001 - -### SSH Port -SSH_PORT=22 - -### 1、DataCheck APPJOINT，This service is used to provide DataCheck capability. -HIVE_META_URL=jdbc:mysql://127.0.0.1:3306/hivemeta?characterEncoding=UTF-8 -HIVE_META_USER=xxx -HIVE_META_PASSWORD=xxx - -#Used to store the azkaban project transformed by DSS -WDS_SCHEDULER_PATH=file:///appcom/tmp/wds/scheduler - -###The IP address and port are written into the database here, so be sure to plan ahead -## visualis-server -VISUALIS_SERVER_INSTALL_IP=127.0.0.1 -VISUALIS_SERVER_PORT=9007 -### visualis nginx acess ip,keep consistent with DSS front end -VISUALIS_NGINX_IP=127.0.0.1 -VISUALIS_NGINX_PORT=8088 - -### Eventchecker APPJOINT -### This service is used to provide Eventchecker capability. it's config in db.sh same as dss-server. - -#azkaban address for check -AZKABAN_ADRESS_IP=127.0.0.1 -AZKABAN_ADRESS_PORT=8081 - -#qualitis.address for check -QUALITIS_ADRESS_IP=127.0.0.1 -QUALITIS_ADRESS_PORT=8090 - -DSS_VERSION=0.9.1 diff --git a/conf/db.sh b/conf/db.sh deleted file mode 100644 index 68b07a9b3..000000000 --- a/conf/db.sh +++ /dev/null @@ -1,8 +0,0 @@ -### for DSS-Server and Eventchecker APPJOINT -MYSQL_HOST=127.0.0.1 -MYSQL_PORT=3306 -MYSQL_DB=xxx -MYSQL_USER=xxx -MYSQL_PASSWORD=xxx - - diff --git a/conf/dss-apiservice-server.properties b/conf/dss-apiservice-server.properties new file mode 100644 index 000000000..5162f42bc --- /dev/null +++ b/conf/dss-apiservice-server.properties @@ -0,0 +1,42 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# Spring configurations +spring.server.port=9206 +spring.spring.application.name=dss-apiservice-server + +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/apiservice/core/dao/mapper/*.xml +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.apiservice.core.bo,com.webank.wedatasphere.dss.apiservice.core.vo +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.apiservice.core.dao + +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.apiservice.core.restful + +#sit +wds.linkis.server.version=v1 +wds.linkis.server.url= + +#test +wds.linkis.test.mode=false +wds.linkis.test.user= + + +#dsm +wds.linkis.server.dsm.admin.users= + + +#用于执行的datasource配置 +wds.linkis.datasource.hikari.maximumPoolSize=100 +wds.linkis.datasource.hikari.minimumIdle=10 diff --git a/conf/dss-datapipe-server.properties b/conf/dss-datapipe-server.properties new file mode 100644 index 000000000..b3d4a8c75 --- /dev/null +++ b/conf/dss-datapipe-server.properties @@ -0,0 +1,33 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# Spring configurations +spring.server.port=9009 +spring.spring.application.name=dss-datapipe-server + +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/datapipe/dao/mapper/*.xml +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.datapipe.vo +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.datapipe.dao + +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.datapipe.restful + +#sit +wds.linkis.server.version=v1 +wds.linkis.server.url= + +#test +wds.linkis.test.mode=false +wds.linkis.test.user= diff --git a/conf/dss-flow-execution-server.properties b/conf/dss-flow-execution-server.properties new file mode 100644 index 000000000..52b151cb3 --- /dev/null +++ b/conf/dss-flow-execution-server.properties @@ -0,0 +1,54 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# Spring configurations +spring.server.port=9205 + +spring.spring.application.name=dss-flow-entrance + +##mybatis +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/flow/execution/entrance/dao/impl/*.xml,classpath*:com/webank/wedatasphere/linkis/jobhistory/dao/impl/*.xml + +wds.linkis.server.mybatis.typeAliasesPackage= + +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.flow.execution.entrance.dao,com.webank.wedatasphere.linkis.jobhistory.dao + + +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.linkis.entrance.restful,com.webank.wedatasphere.dss.flow.execution.entrance.restful + +#wds.linkis.server.component.exclude.classes=com.webank.wedatasphere.linkis.DataWorkCloudApplication + +wds.linkis.engine.application.name=flowExecutionEngine +wds.linkis.enginemanager.application.name=flowExecution + +wds.linkis.query.application.name=linkis-ps-publicservice + +wds.linkis.console.config.application.name=linkis-ps-publicservice +wds.linkis.engine.creation.wait.time.max=20m +wds.linkis.server.version=v1 + +wds.linkis.server.socket.mode=true + +wds.linkis.client.flow.adminuser=ws +wds.linkis.client.flow.author.user.token=WS-AUTH + +wds.linkis.server.component.exclude.classes=com.webank.wedatasphere.linkis.entranceclient.conf.ClientForEntranceSpringConfiguration,com.webank.wedatasphere.linkis.entranceclient.conf.ClientSpringConfiguration + +wds.linkis.server.component.exclude.packages=com.webank.wedatasphere.linkis.entrance.restful. +spring.spring.main.allow-bean-definition-overriding=true + +wds.linkis.entrance.config.log.path=file:///appcom/tmp/dss/ + diff --git a/conf/dss-framework-orchestrator-server.properties b/conf/dss-framework-orchestrator-server.properties new file mode 100644 index 000000000..c1f80dc56 --- /dev/null +++ b/conf/dss-framework-orchestrator-server.properties @@ -0,0 +1,42 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# Spring configurations +spring.server.port=9203 +spring.spring.application.name=dss-framework-orchestrator-server + +wds.linkis.test.mode=true + +wds.linkis.test.user=neiljianliu + +wds.linkis.log.clear=true + +wds.linkis.server.version=v1 + +##restful +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.orchestrator.server.restful + +##mybatis +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/framework/appconn/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/orchestrator/core/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/server/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/application/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/workspace/mapper/impl/*.xml,classpath*:com/webank/wedatasphere/dss/workspace/common/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/orchestrator/db/dao/impl/*.xml + +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.server.entity,com.webank.wedatasphere.dss.application.entity,com.webank.wedatasphere.dss.framework.appconn.entity + +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.framework.appconn.dao,com.webank.wedatasphere.dss.orchestrator.core.dao,com.webank.wedatasphere.dss.server.dao,com.webank.wedatasphere.dss.application.dao,com.webank.wedatasphere.dss.workspace.mapper,com.webank.wedatasphere.dss.workspace.common.dao,com.webank.wedatasphere.dss.workspace.common.dao,com.webank.wedatasphere.dss.orchestrator.db.dao + +wds.dss.appconn.scheduler.project.store.dir=file:///appcom/tmp/wds/scheduler +wds.dss.appconn.scheduler.azkaban.login.passwd=userpwd +##export file dir +wds.dss.server.export.url=/appcom/tmp/dss diff --git a/conf/dss-framework-project-server.properties b/conf/dss-framework-project-server.properties new file mode 100644 index 000000000..a4d2356d0 --- /dev/null +++ b/conf/dss-framework-project-server.properties @@ -0,0 +1,35 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# Spring configurations +spring.server.port=9202 +spring.spring.application.name=dss-framework-project-server + +wds.linkis.log.clear=true + +wds.linkis.server.version=v1 + +##restful +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.framework.workspace.restful,com.webank.wedatasphere.dss.framework.project.restful,com.webank.wedatasphere.dss.framework.release.restful,com.webank.wedatasphere.dss.framework.appconn.restful + +##mybatis +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/framework/workspace/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/application/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/framework/project/dao/impl/*Mapper.xml,classpath*:com/webank/wedatasphere/dss/framework/appconn/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/framework/release/dao/impl/*.xml + +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.application.entity,com.webank.wedatasphere.dss.common.entity,com.webank.wedatasphere.dss.framework.workspace.bean,com.webank.wedatasphere.dss.framework.project.entity,com.webank.wedatasphere.dss.framework.appconn.entity,com.webank.wedatasphere.dss.framework.release.entity + +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.framework.workspace.dao,com.webank.wedatasphere.dss.application.dao,com.webank.wedatasphere.dss.framework.project.dao,com.webank.wedatasphere.dss.framework.appconn.dao,com.webank.wedatasphere.dss.framework.release.dao + + diff --git a/conf/dss-workflow-server.properties b/conf/dss-workflow-server.properties new file mode 100644 index 000000000..d782c9a35 --- /dev/null +++ b/conf/dss-workflow-server.properties @@ -0,0 +1,44 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# Spring configurations +spring.server.port=9207 +spring.spring.application.name=dss-workflow-server + +wds.linkis.test.mode=true + +wds.linkis.test.user=neiljianliu + +wds.linkis.log.clear=true + +wds.linkis.server.version=v1 + +##restful +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.workflow.restful + +##mybatis +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/workflow/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/framework/appconn/dao/impl/*.xml + +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.workflow.entity,com.webank.wedatasphere.dss.framework.appconn.entity + +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.workflow.dao,com.webank.wedatasphere.dss.framework.appconn.dao + +wds.dss.appconn.scheduler.project.store.dir=file:///appcom/tmp/wds/scheduler +wds.dss.appconn.scheduler.azkaban.login.passwd= +##import file dir +wds.dss.file.upload.dir=/appcom/tmp/uploads + +wds.dss.server.export.env=DEV \ No newline at end of file diff --git a/conf/dss.properties b/conf/dss.properties new file mode 100644 index 000000000..e381e4054 --- /dev/null +++ b/conf/dss.properties @@ -0,0 +1,30 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +wds.linkis.gateway.ip=127.0.0.1 +wds.linkis.gateway.port=9001 +wds.linkis.gateway.url=http://127.0.0.1:9001/ +wds.linkis.gateway.wtss.url=http://127.0.0.1:9001/ + +wds.linkis.mysql.is.encrypt=false +wds.linkis.server.mybatis.datasource.url=jdbc:mysql://127.0.0.1:3306/dss_dev?characterEncoding=UTF-8 +wds.linkis.server.mybatis.datasource.username= +***REMOVED*** + +wds.dss.esb.appid= +wds.dss.esb.token= + +wds.dss.appconn.scheduler.job.label=dev \ No newline at end of file diff --git a/conf/log4j.properties b/conf/log4j.properties new file mode 100644 index 000000000..c94a1c841 --- /dev/null +++ b/conf/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/dss-apiservice-server.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO \ No newline at end of file diff --git a/conf/log4j2.xml b/conf/log4j2.xml new file mode 100644 index 000000000..8e22a9320 --- /dev/null +++ b/conf/log4j2.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/conf/token.properties b/conf/token.properties new file mode 100644 index 000000000..5674d0245 --- /dev/null +++ b/conf/token.properties @@ -0,0 +1,17 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +${userName}=${password} \ No newline at end of file diff --git a/datachecker-appjoint/pom.xml b/datachecker-appjoint/pom.xml deleted file mode 100644 index d609c3ab3..000000000 --- a/datachecker-appjoint/pom.xml +++ /dev/null @@ -1,139 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-datachecker-appjoint - - - - org.apache.commons - commons-lang3 - 3.4 - - - - com.alibaba - druid - 1.0.28 - - - - com.webank.wedatasphere.dss - dss-appjoint-core - ${dss.version} - provided - true - - - - - - - - - - - - - - - - log4j - log4j - 1.2.17 - - - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - dss-datachecker-appjoint - false - false - - src/main/assembly/distribution.xml - - - - - - - src/main/java - - **/*.xml - - - - src/main/resources - - **/*.properties - **/application.yml - **/bootstrap.yml - **/log4j2.xml - - - - - - - \ No newline at end of file diff --git a/datachecker-appjoint/src/main/assembly/distribution.xml b/datachecker-appjoint/src/main/assembly/distribution.xml deleted file mode 100644 index 503ab0f38..000000000 --- a/datachecker-appjoint/src/main/assembly/distribution.xml +++ /dev/null @@ -1,136 +0,0 @@ - - - - dss-datachecker-appjoint - - zip - - true - datachecker - - - - - - lib - true - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ${basedir}/src/main/resources - - appjoint.properties - - 0777 - / - unix - - - - ${basedir}/src/main/resources - - log4j.properties - log4j2.xml - - 0777 - conf - unix - - - - - - diff --git a/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/DataCheckerDao.java b/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/DataCheckerDao.java deleted file mode 100644 index d84a0d4bc..000000000 --- a/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/DataCheckerDao.java +++ /dev/null @@ -1,195 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.connector; - -import com.alibaba.druid.pool.DruidDataSource; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.DataChecker; -import org.apache.log4j.Logger; - -import javax.sql.DataSource; -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Properties; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import java.util.stream.Collectors; - -public class DataCheckerDao { - - private static final String SQL_SOURCE_TYPE_JOB_TABLE = - "SELECT * FROM DBS d JOIN TBLS t ON t.DB_ID = d.DB_ID WHERE d.NAME=? AND t.TBL_NAME=?"; - - private static final String SQL_SOURCE_TYPE_JOB_PARTITION = - "SELECT * FROM DBS d JOIN TBLS t ON t.DB_ID = d.DB_ID JOIN PARTITIONS p ON p.TBL_ID = t.TBL_ID WHERE d.NAME=? AND t.TBL_NAME=? AND p.PART_NAME=?"; - - private static DataSource jobDS; - private static DataCheckerDao instance; - - public static DataCheckerDao getInstance() { - if (instance == null) { - synchronized (DataCheckerDao.class) { - if (instance == null) { - instance = new DataCheckerDao(); - } - } - } - return instance; - } - - public boolean validateTableStatusFunction(Properties props, Logger log) { - if (jobDS == null) { - jobDS = DataDruidFactory.getJobInstance(props, log); - if (jobDS == null) { - log.error("Error getting Druid DataSource instance"); - return false; - } - } - removeBlankSpace(props); - log.info("=============================Data Check Start=========================================="); - String dataCheckerInfo = props.getProperty(DataChecker.DATA_OBJECT); - log.info("(DataChecker info) database table partition info : " + dataCheckerInfo); - long waitTime = Long.valueOf(props.getProperty(DataChecker.WAIT_TIME, "1")) * 3600 * 1000; - int queryFrequency = Integer.valueOf(props.getProperty(DataChecker.QUERY_FREQUENCY, "30000")); -// String timeScape = props.getProperty(DataChecker.TIME_SCAPE, "NULL"); - log.info("(DataChecker info) wait time : " + waitTime); - log.info("(DataChecker info) query frequency : " + queryFrequency); -// log.info("(DataChecker info) time scape : " + timeScape); - List> dataObjectList = extractProperties(props); - try (Connection conn = jobDS.getConnection()) { - boolean flag = dataObjectList - .stream() - .allMatch(proObjectMap -> { - long count = getTotalCount(proObjectMap, conn, log); - return count > 0; - }); - if (flag){ - log.info("=============================Data Check End=========================================="); - return true; - } - - } catch (SQLException e) { - throw new RuntimeException("get DataChecker result failed", e); - } - - log.info("=============================Data Check End=========================================="); - return false; - } - - private void sleep(long sleepTime) { - try { - Thread.sleep(sleepTime); - } catch (InterruptedException e) { - e.printStackTrace(); - } - } - - private void removeBlankSpace(Properties props) { - try { - props.entrySet().forEach(entry -> { - String value = entry.getValue().toString().replaceAll(" ", "").trim(); - entry.setValue(value); - }); - }catch (Exception e){ - throw new RuntimeException("remove job space char failed",e); - } - } - - private List> extractProperties(Properties p) { - return p.keySet().stream() - .map(key -> key2Map(key, p)).filter(x -> x.size() >0) - .collect(Collectors.toList()); - } - - private Map key2Map(Object key, Properties p) { - Map proMap = new HashMap<>(); - String skey = String.valueOf(key); - if(skey.contains(DataChecker.DATA_OBJECT)){ - String[] keyArr = skey.split("\\."); - if(keyArr.length == 3){ - String keyNum = keyArr[2]; - String doKey = DataChecker.DATA_OBJECT + "." + keyNum; - proMap.put(DataChecker.DATA_OBJECT, String.valueOf(p.get(doKey))); - }else{ - String doKey = DataChecker.DATA_OBJECT; - proMap.put(DataChecker.DATA_OBJECT, String.valueOf(p.get(doKey))); - } - } - return proMap; - } - - private long getTotalCount(Map proObjectMap, Connection conn, Logger log) { - String dataObject = proObjectMap.get(DataChecker.DATA_OBJECT); - if(dataObject != null) { - dataObject = dataObject.replace(" ", "").trim(); - }else{ - log.error("DataObject is null"); - return 0; - } - log.info("-------------------------------------- search hive/spark/mr data "); - log.info("-------------------------------------- : " + dataObject); - try (PreparedStatement pstmt = getStatement(conn, dataObject)) { - ResultSet rs = pstmt.executeQuery(); - return rs.last() ? rs.getRow() : 0; - } catch (SQLException e) { - log.error("fetch data from Hive MetaStore error", e); - return 0; - } - } - - private PreparedStatement getStatement(Connection conn, String dataObject) throws SQLException { - String dataScape = dataObject.contains("{") ? "Partition" : "Table"; - String[] dataObjectArray = dataObject.split("\\."); - String dbName = dataObjectArray[0]; - String tableName = dataObjectArray[1]; - if(dataScape.equals("Partition")) { - Pattern pattern = Pattern.compile("\\{([^\\}]+)\\}"); - Matcher matcher = pattern.matcher(dataObject); - String partitionName = null; - if(matcher.find()){ - partitionName = matcher.group(1); - } - partitionName = partitionName.replace("\'", "").replace("\"", ""); - tableName = tableName.split("\\{")[0]; - PreparedStatement pstmt = conn.prepareCall(SQL_SOURCE_TYPE_JOB_PARTITION); - pstmt.setString(1, dbName); - pstmt.setString(2, tableName); - pstmt.setString(3, partitionName); - return pstmt; - } else if(dataObjectArray.length == 2){ - PreparedStatement pstmt = conn.prepareCall(SQL_SOURCE_TYPE_JOB_TABLE); - pstmt.setString(1, dbName); - pstmt.setString(2, tableName); - return pstmt; - }else { - throw new SQLException("Incorrect input format for dataObject "+ dataObject); - } - } - - public static void closeDruidDataSource(){ - DruidDataSource jobDSObject = (DruidDataSource)jobDS; - if(jobDSObject != null){ - jobDSObject.close(); - } - } - -} diff --git a/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/DataDruidFactory.java b/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/DataDruidFactory.java deleted file mode 100644 index b55868125..000000000 --- a/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/DataDruidFactory.java +++ /dev/null @@ -1,122 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.connector; - -import com.alibaba.druid.pool.DruidDataSource; -import org.apache.commons.lang3.StringUtils; -import org.apache.log4j.Logger; - -import java.util.Base64; -import java.util.Properties; - -public class DataDruidFactory { - private static DruidDataSource jobInstance; - private static DruidDataSource msgInstance; - - public static DruidDataSource getJobInstance(Properties props, Logger log) { - if (jobInstance == null ) { - synchronized (DataDruidFactory.class) { - if(jobInstance == null) { - try { - jobInstance = createDataSource(props, log, "Job"); - } catch (Exception e) { - throw new RuntimeException("Error creating Druid DataSource", e); - } - } - } - } - return jobInstance; - } - - public static DruidDataSource getMsgInstance(Properties props, Logger log) { - if (msgInstance == null ) { - synchronized (DataDruidFactory.class) { - if(msgInstance == null) { - try { - msgInstance = createDataSource(props, log, "Msg"); - } catch (Exception e) { - throw new RuntimeException("Error creating Druid DataSource", e); - } - } - } - } - return msgInstance; - } - - private static DruidDataSource createDataSource(Properties props, Logger log, String type) { - String name = null; - String url = null; - String username = null; - String password = null; - - if (type.equals("Job")) { - name = props.getProperty("job.datachecker.jdo.option.name"); - url = props.getProperty("job.datachecker.jdo.option.url"); - username = props.getProperty("job.datachecker.jdo.option.username"); - password = props.getProperty("job.datachecker.jdo.option.password"); - try { -// password = new String(Base64.getDecoder().decode(props.getProperty("job.datachecker.jdo.option.password").getBytes()),"UTF-8"); - password = props.getProperty("job.datachecker.jdo.option.password"); - } catch (Exception e){ - log.error("password decore failed" + e); - } - } - int initialSize = Integer.valueOf(props.getProperty("datachecker.jdo.option.initial.size", "1")); - int maxActive = Integer.valueOf(props.getProperty("datachecker.jdo.option.max.active", "100")); - int minIdle = Integer.valueOf(props.getProperty("datachecker.jdo.option.min.idle", "1")); - long maxWait = Long.valueOf(props.getProperty("datachecker.jdo.option.max.wait", "60000")); - String validationQuery = props.getProperty("datachecker.jdo.option.validation.quert", "SELECT 'x'"); - long timeBetweenEvictionRunsMillis = Long.valueOf(props.getProperty("datachecker.jdo.option.time.between.eviction.runs.millis", "6000")); - long minEvictableIdleTimeMillis = Long.valueOf(props.getProperty("datachecker.jdo.option.evictable.idle,time.millis", "300000")); - boolean testOnBorrow = Boolean.valueOf(props.getProperty("datachecker.jdo.option.test.on.borrow", "true")); - int maxOpenPreparedStatements = Integer.valueOf(props.getProperty("datachecker.jdo.option.max.open.prepared.statements", "-1")); - - - if (timeBetweenEvictionRunsMillis > minEvictableIdleTimeMillis) { - timeBetweenEvictionRunsMillis = minEvictableIdleTimeMillis; - } - - DruidDataSource ds = new DruidDataSource(); - - if (StringUtils.isNotBlank(name)) { - ds.setName(name); - } - - ds.setUrl(url); - ds.setDriverClassName("com.mysql.jdbc.Driver"); - ds.setUsername(username); - ds.setPassword(password); - ds.setInitialSize(initialSize); - ds.setMinIdle(minIdle); - ds.setMaxActive(maxActive); - ds.setMaxWait(maxWait); - ds.setTestOnBorrow(testOnBorrow); - ds.setValidationQuery(validationQuery); - ds.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis); - ds.setMinEvictableIdleTimeMillis(minEvictableIdleTimeMillis); - if (maxOpenPreparedStatements > 0) { - ds.setPoolPreparedStatements(true); - ds.setMaxPoolPreparedStatementPerConnectionSize( - maxOpenPreparedStatements); - } else { - ds.setPoolPreparedStatements(false); - } - log.info("Druid data source initialed!"); - return ds; - } -} diff --git a/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/entity/DataChecker.java b/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/entity/DataChecker.java deleted file mode 100644 index b108cc6b9..000000000 --- a/datachecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/entity/DataChecker.java +++ /dev/null @@ -1,90 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity; - -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionState; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.connector.DataCheckerDao; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution.DataCheckerNodeExecutionAction; -import org.apache.log4j.Logger; - -import java.util.Properties; - -public class DataChecker { - public final static String SOURCE_TYPE = "source.type"; - public final static String DATA_OBJECT = "check.object"; - public final static String WAIT_TIME = "max.check.hours"; - public final static String QUERY_FREQUENCY = "query.frequency"; - public final static String TIME_SCAPE = "time.scape"; - - private Properties p; - private static final Logger logger = Logger.getRootLogger(); - DataCheckerDao wbDao = DataCheckerDao.getInstance(); - DataCheckerNodeExecutionAction dataCheckerAction = null; - public long maxWaitTime; - public int queryFrequency; - - public DataChecker(String jobName, Properties p,DataCheckerNodeExecutionAction action) { - this.p = p; - dataCheckerAction = action; - maxWaitTime = Long.valueOf(p.getProperty(DataChecker.WAIT_TIME, "1")) * 3600 * 1000; - queryFrequency = Integer.valueOf(p.getProperty(DataChecker.QUERY_FREQUENCY, "30000")); - - } - - public void run() { - dataCheckerAction.setState(NodeExecutionState.Running); - try { - if(p == null) { - throw new RuntimeException("Properties is null. Can't continue"); - } - if (!p.containsKey(SOURCE_TYPE)) { - logger.info("Properties " + SOURCE_TYPE + " value is Null !"); - } - if (!p.containsKey(DATA_OBJECT)) { - logger.info("Properties " + DATA_OBJECT + " value is Null !"); - } - begineCheck(); - }catch (Exception ex){ - dataCheckerAction.setState(NodeExecutionState.Failed); - throw new RuntimeException("get DataChecker result failed", ex); - } - - } - - public void begineCheck(){ - boolean success=false; - try { - success= wbDao.validateTableStatusFunction(p, logger); - }catch (Exception ex){ - dataCheckerAction.setState(NodeExecutionState.Failed); - logger.error("datacheck error",ex); - throw new RuntimeException("get DataChecker result failed", ex); - } - if(success) { - dataCheckerAction.setState(NodeExecutionState.Success); - }else { - dataCheckerAction.setState(NodeExecutionState.Running); - } - } - - public void cancel() { -// DataCheckerDao.closeDruidDataSource(); -// throw new RuntimeException("Kill this DataChecker job."); - } - -} \ No newline at end of file diff --git a/datachecker-appjoint/src/main/resources/appjoint.properties b/datachecker-appjoint/src/main/resources/appjoint.properties deleted file mode 100644 index 01e9ff0b4..000000000 --- a/datachecker-appjoint/src/main/resources/appjoint.properties +++ /dev/null @@ -1,24 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -job.datachecker.jdo.option.name=job -job.datachecker.jdo.option.url=jdbc:mysql://127.0.0.1:3306/ -job.datachecker.jdo.option.username= -job.datachecker.jdo.option.password= - - - diff --git a/datachecker-appjoint/src/main/resources/log4j.properties b/datachecker-appjoint/src/main/resources/log4j.properties deleted file mode 100644 index 0807e6087..000000000 --- a/datachecker-appjoint/src/main/resources/log4j.properties +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO diff --git a/datachecker-appjoint/src/main/resources/log4j2.xml b/datachecker-appjoint/src/main/resources/log4j2.xml deleted file mode 100644 index 3923cd9f3..000000000 --- a/datachecker-appjoint/src/main/resources/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerAppJoint.scala b/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerAppJoint.scala deleted file mode 100644 index f71b8307d..000000000 --- a/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerAppJoint.scala +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution - -import java.util - -import com.webank.wedatasphere.dss.appjoint.AppJoint -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl - -/** - * Created by enjoyyin on 2019/11/5. - */ -class DataCheckerAppJoint extends AppJointUrlImpl with AppJoint { - - private var params: util.Map[String, AnyRef] = _ - private var nodeExecution: NodeExecution = _ - - override def getAppJointName: String = "DataChecker" - - override def init(baseUrl: String, params: util.Map[String, AnyRef]): Unit = { - setBaseUrl(baseUrl) - this.params = params - } - - override def getNodeExecution: NodeExecution = { - if(nodeExecution == null) synchronized { - if(nodeExecution == null) { - nodeExecution = new DataCheckerExecution() - nodeExecution.setBaseUrl(getBaseUrl) - nodeExecution.init(params) - } - } - nodeExecution - } -} diff --git a/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerExecution.scala b/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerExecution.scala deleted file mode 100644 index c169d4431..000000000 --- a/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerExecution.scala +++ /dev/null @@ -1,157 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution - -import java.util -import java.util.{Properties, UUID} - -import com.webank.wedatasphere.dss.appjoint.execution.common._ -import com.webank.wedatasphere.dss.appjoint.execution.core._ -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.DataChecker -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrl -import com.webank.wedatasphere.dss.appjoint.service.session.Session -import com.webank.wedatasphere.linkis.common.utils.Utils -import org.slf4j.LoggerFactory; - -/** - * Created by allenlliu on 2019/11/11. - */ -class DataCheckerExecution extends LongTermNodeExecution with AppJointUrl with Killable with Procedure { - private val logger = LoggerFactory.getLogger(classOf[DataCheckerExecution]) - import scala.collection.JavaConversions.mapAsScalaMap - var appJointParams : scala.collection.mutable.Map[String,AnyRef]= null - - - /** - * 表示任务能否提交到该AppJoint去执行 - * - * @param node AppJointNode - * @return true is ok while false is not - */ - override def canExecute(node: AppJointNode, context: NodeContext, session: Session): Boolean = node.getNodeType.toLowerCase.contains("datachecker") - - protected def putErrorMsg(errorMsg: String, t: Throwable, action: DataCheckerNodeExecutionAction): DataCheckerNodeExecutionAction = t match { - - case t: Exception => - val response = action.response - response.setErrorMsg(errorMsg) - response.setException(t) - response.setIsSucceed(false) - action - } - - override def init(params: util.Map[String, AnyRef]): Unit = { - this.appJointParams = params - } - override def submit(node: AppJointNode, context: NodeContext, session:Session): NodeExecutionAction = { - val nodeAction = new DataCheckerNodeExecutionAction() - nodeAction.setId(UUID.randomUUID().toString()) - val jobName = node.getName - val scalaParams: scala.collection.mutable.Map[String,Object] =context.getRuntimeMap - val properties = new Properties() - this.appJointParams.foreach{ - case (key: String, value: Object) => - logger.info("appjoint params key : "+key+",value : "+value) - properties.put(key, value.toString) - } - scalaParams.foreach { case (key: String, value: Object) => - logger.info("request params key : "+key+",value : "+value) - properties.put(key, value.toString) - } - val dc = new DataChecker(jobName,properties,nodeAction) - dc.run() - nodeAction.setDc(dc) - nodeAction - } - - override def state(action: NodeExecutionAction): NodeExecutionState = { - action match { - case action: DataCheckerNodeExecutionAction => { - if (action.state.isCompleted) return action.state - action.dc.begineCheck() - action.state - } - case _ => NodeExecutionState.Failed - } - } - private var baseUrl:String ="" - - override def getBaseUrl: String = baseUrl - - override def setBaseUrl(basicUrl: String): Unit = { - this.baseUrl = basicUrl - } - - override def result(action: NodeExecutionAction, nodeContext: NodeContext): CompletedNodeExecutionResponse = { - val response = new CompletedNodeExecutionResponse - action match { - case action: DataCheckerNodeExecutionAction => { - if (action.state.equals(NodeExecutionState.Success)) { - response.setIsSucceed(true) - } else { - response.setIsSucceed(false) - } - response - } - case _ => { - response.setIsSucceed(false) - response - } - - - } - } - - override def kill(action: NodeExecutionAction): Boolean = action match { - case longTermAction: DataCheckerNodeExecutionAction => - getScheduler.removeAsyncResponse(longTermAction) - true - } - - override def progress(action: NodeExecutionAction): Float = { - return 0.5f - } - - override def log(action: NodeExecutionAction): String = { - action match { - case action: DataCheckerNodeExecutionAction => { - if (!action.state.isCompleted) { - "DataChecker is waiting for tables" - } else { - "DataChecker successfully received info of tables" - } - } - case _ => "Error for NodeExecutionAction " - } - - } - - override def createAsyncNodeExecutionResponse(node: AppJointNode, context: NodeContext, action: NodeExecutionAction): AsyncNodeExecutionResponse = { - action match { - case action: DataCheckerNodeExecutionAction => { - val response = new AsyncNodeExecutionResponse - response.setAction(action) - response.setAppJointNode(node) - response.setNodeContext(context) - response.setMaxLoopTime(action.dc.maxWaitTime) - response.setAskStatePeriod(action.dc.queryFrequency) - response - } - } - } -} diff --git a/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerNodeExecutionAction.scala b/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerNodeExecutionAction.scala deleted file mode 100644 index fff35e067..000000000 --- a/datachecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/DataCheckerNodeExecutionAction.scala +++ /dev/null @@ -1,46 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution - -import com.webank.wedatasphere.dss.appjoint.execution.common.{AbstractNodeExecutionAction, CompletedNodeExecutionResponse, LongTermNodeExecutionAction, NodeExecutionState} -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.DataChecker - -/** - * Created by allenlliu on 2019/11/12. - */ -class DataCheckerNodeExecutionAction extends AbstractNodeExecutionAction with LongTermNodeExecutionAction { - private[this] var _state: NodeExecutionState = null - private var schedulerId: Int = _ - def state: NodeExecutionState = _state - - def setState(value: NodeExecutionState): Unit = { - _state = value - } - val response = new CompletedNodeExecutionResponse - private[this] var _dc: DataChecker = null - - def dc: DataChecker = _dc - - def setDc(value: DataChecker): Unit = { - _dc = value - } - - override def setSchedulerId(schedulerId: Int): Unit = this.schedulerId = schedulerId - - override def getSchedulerId: Int = schedulerId -} diff --git a/db/azkaban.sql b/db/azkaban.sql deleted file mode 100644 index 7f18b3308..000000000 --- a/db/azkaban.sql +++ /dev/null @@ -1,4 +0,0 @@ -INSERT INTO `dss_application` (`id`, `name`, `url`, `is_user_need_init`, `level`, `user_init_url`, `exists_project_service`, `project_url`, `enhance_json`, `if_iframe`, `homepage_url`, `redirect_url`) VALUES (NULL, 'schedulis', NULL, '0', '1', NULL, '0', NULL, NULL, '1', NULL, NULL); -UPDATE `dss_application` SET url = 'http://AZKABAN_ADRESS_IP_2:AZKABAN_ADRESS_PORT', project_url = 'http://AZKABAN_ADRESS_IP_2:AZKABAN_ADRESS_PORT/manager?project=${projectName}',homepage_url = 'http://AZKABAN_ADRESS_IP_2:AZKABAN_ADRESS_PORT/homepage' WHERE `name` in ('schedulis'); -SELECT @shcedulis_id:=id FROM `dss_application` WHERE `name` = 'schedulis'; -insert into dss_workflow_node values(null,null,'linkis.shell.sh',@shcedulis_id,1,1,0,1,null); diff --git a/db/davinci.sql b/db/davinci.sql deleted file mode 100644 index 29ed90a7a..000000000 --- a/db/davinci.sql +++ /dev/null @@ -1,666 +0,0 @@ -SET NAMES utf8mb4; -SET FOREIGN_KEY_CHECKS = 0; - --- ---------------------------- --- Table structure for cron_job --- ---------------------------- -DROP TABLE IF EXISTS `cron_job`; -CREATE TABLE `cron_job` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(45) COLLATE utf8_unicode_ci NOT NULL, - `project_id` bigint(20) NOT NULL, - `job_type` varchar(45) COLLATE utf8_unicode_ci NOT NULL, - `job_status` varchar(10) COLLATE utf8_unicode_ci NOT NULL DEFAULT '', - `cron_expression` varchar(45) COLLATE utf8_unicode_ci NOT NULL, - `start_date` datetime NOT NULL, - `end_date` datetime NOT NULL, - `config` text COLLATE utf8_unicode_ci NOT NULL, - `description` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL, - `exec_log` text COLLATE utf8_unicode_ci, - `create_by` bigint(20) NOT NULL, - `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, - `update_time` timestamp NULL DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `name_UNIQUE` (`name`) USING BTREE -) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; - --- ---------------------------- --- Table structure for dashboard --- ---------------------------- -DROP TABLE IF EXISTS `dashboard`; -CREATE TABLE `dashboard` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `dashboard_portal_id` bigint(20) NOT NULL, - `type` smallint(1) NOT NULL, - `index` int(4) NOT NULL, - `parent_id` bigint(20) NOT NULL DEFAULT '0', - `config` text, - `full_parent_Id` varchar(100) DEFAULT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_dashboard_id` (`dashboard_portal_id`) USING BTREE, - KEY `idx_parent_id` (`parent_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for dashboard_portal --- ---------------------------- -DROP TABLE IF EXISTS `dashboard_portal`; -CREATE TABLE `dashboard_portal` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `project_id` bigint(20) NOT NULL, - `avatar` varchar(255) DEFAULT NULL, - `publish` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_project_id` (`project_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for display --- ---------------------------- -DROP TABLE IF EXISTS `display`; -CREATE TABLE `display` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `project_id` bigint(20) NOT NULL, - `avatar` varchar(255) DEFAULT NULL, - `publish` tinyint(1) NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_project_id` (`project_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for display_slide --- ---------------------------- -DROP TABLE IF EXISTS `display_slide`; -CREATE TABLE `display_slide` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `display_id` bigint(20) NOT NULL, - `index` int(12) NOT NULL, - `config` text NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_display_id` (`display_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for download_record --- ---------------------------- -DROP TABLE IF EXISTS `download_record`; -CREATE TABLE `download_record` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `user_id` bigint(20) NOT NULL, - `path` varchar(255) DEFAULT NULL, - `status` smallint(1) NOT NULL, - `create_time` datetime NOT NULL, - `last_download_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_user` (`user_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for favorite --- ---------------------------- -DROP TABLE IF EXISTS `favorite`; -CREATE TABLE `favorite` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `user_id` bigint(20) NOT NULL, - `project_id` bigint(20) NOT NULL, - `create_time` datetime NOT NULL ON UPDATE CURRENT_TIMESTAMP, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `idx_user_project` (`user_id`, `project_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for mem_dashboard_widget --- ---------------------------- -DROP TABLE IF EXISTS `mem_dashboard_widget`; -CREATE TABLE `mem_dashboard_widget` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `dashboard_id` bigint(20) NOT NULL, - `widget_Id` bigint(20) DEFAULT NULL, - `x` int(12) NOT NULL, - `y` int(12) NOT NULL, - `width` int(12) NOT NULL, - `height` int(12) NOT NULL, - `polling` tinyint(1) NOT NULL DEFAULT '0', - `frequency` int(12) DEFAULT NULL, - `config` text, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_protal_id` (`dashboard_id`) USING BTREE, - KEY `idx_widget_id` (`widget_Id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for mem_display_slide_widget --- ---------------------------- -DROP TABLE IF EXISTS `mem_display_slide_widget`; -CREATE TABLE `mem_display_slide_widget` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `display_slide_id` bigint(20) NOT NULL, - `widget_id` bigint(20) DEFAULT NULL, - `name` varchar(255) NOT NULL, - `params` text NOT NULL, - `type` smallint(1) NOT NULL, - `sub_type` smallint(2) DEFAULT NULL, - `index` int(12) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_slide_id` (`display_slide_id`) USING BTREE, - KEY `idx_widget_id` (`widget_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for organization --- ---------------------------- -DROP TABLE IF EXISTS `organization`; -CREATE TABLE `organization` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `avatar` varchar(255) DEFAULT NULL, - `user_id` bigint(20) NOT NULL, - `project_num` int(20) DEFAULT '0', - `member_num` int(20) DEFAULT '0', - `role_num` int(20) DEFAULT '0', - `allow_create_project` tinyint(1) DEFAULT '1', - `member_permission` smallint(1) NOT NULL DEFAULT '0', - `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, - `create_by` bigint(20) NOT NULL DEFAULT '0', - `update_time` timestamp NULL DEFAULT NULL, - `update_by` bigint(20) DEFAULT '0', - PRIMARY KEY (`id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for platform --- ---------------------------- -DROP TABLE IF EXISTS `platform`; -CREATE TABLE `platform` -( - `id` bigint(20) NOT NULL, - `name` varchar(255) NOT NULL, - `platform` varchar(255) NOT NULL, - `code` varchar(32) NOT NULL, - `checkCode` varchar(255) DEFAULT NULL, - `checkSystemToken` varchar(255) DEFAULT NULL, - `checkUrl` varchar(255) DEFAULT NULL, - `alternateField1` varchar(255) DEFAULT NULL, - `alternateField2` varchar(255) DEFAULT NULL, - `alternateField3` varchar(255) DEFAULT NULL, - `alternateField4` varchar(255) DEFAULT NULL, - `alternateField5` varchar(255) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for project --- ---------------------------- -DROP TABLE IF EXISTS `project`; -CREATE TABLE `project` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `pic` varchar(255) DEFAULT NULL, - `org_id` bigint(20) NOT NULL, - `user_id` bigint(20) NOT NULL, - `visibility` tinyint(1) DEFAULT '1', - `star_num` int(11) DEFAULT '0', - `is_transfer` tinyint(1) NOT NULL DEFAULT '0', - `initial_org_id` bigint(20) NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for rel_project_admin --- ---------------------------- -DROP TABLE IF EXISTS `rel_project_admin`; -CREATE TABLE `rel_project_admin` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `project_id` bigint(20) NOT NULL, - `user_id` bigint(20) NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `idx_project_user` (`project_id`, `user_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4 COMMENT ='project admin表'; - --- ---------------------------- --- Table structure for rel_role_dashboard --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_dashboard`; -CREATE TABLE `rel_role_dashboard` -( - `role_id` bigint(20) NOT NULL, - `dashboard_id` bigint(20) NOT NULL, - `visible` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`role_id`, `dashboard_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for rel_role_display --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_display`; -CREATE TABLE `rel_role_display` -( - `role_id` bigint(20) NOT NULL, - `display_id` bigint(20) NOT NULL, - `visible` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`role_id`, `display_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for rel_role_portal --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_portal`; -CREATE TABLE `rel_role_portal` -( - `role_id` bigint(20) NOT NULL, - `portal_id` bigint(20) NOT NULL, - `visible` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`role_id`, `portal_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for rel_role_project --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_project`; -CREATE TABLE `rel_role_project` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `project_id` bigint(20) NOT NULL, - `role_id` bigint(20) NOT NULL, - `source_permission` smallint(1) NOT NULL DEFAULT '1', - `view_permission` smallint(1) NOT NULL DEFAULT '1', - `widget_permission` smallint(1) NOT NULL DEFAULT '1', - `viz_permission` smallint(1) NOT NULL DEFAULT '1', - `schedule_permission` smallint(1) NOT NULL DEFAULT '1', - `share_permission` tinyint(1) NOT NULL DEFAULT '0', - `download_permission` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `idx_role_project` (`project_id`, `role_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for rel_role_slide --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_slide`; -CREATE TABLE `rel_role_slide` -( - `role_id` bigint(20) NOT NULL, - `slide_id` bigint(20) NOT NULL, - `visible` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`role_id`, `slide_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for rel_role_user --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_user`; -CREATE TABLE `rel_role_user` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `user_id` bigint(20) NOT NULL, - `role_id` bigint(20) NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `idx_role_user` (`user_id`, `role_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - --- ---------------------------- --- Table structure for rel_role_view --- ---------------------------- -DROP TABLE IF EXISTS `rel_role_view`; -CREATE TABLE `rel_role_view` -( - `view_id` bigint(20) NOT NULL, - `role_id` bigint(20) NOT NULL, - `row_auth` text, - `column_auth` text, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`view_id`, `role_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for rel_user_organization --- ---------------------------- -DROP TABLE IF EXISTS `rel_user_organization`; -CREATE TABLE `rel_user_organization` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `org_id` bigint(20) NOT NULL, - `user_id` bigint(20) NOT NULL, - `role` smallint(1) NOT NULL DEFAULT '0', - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `idx_org_user` (`org_id`, `user_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for role --- ---------------------------- -DROP TABLE IF EXISTS `role`; -CREATE TABLE `role` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `org_id` bigint(20) NOT NULL, - `name` varchar(100) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - `avatar` varchar(255) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_orgid` (`org_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4 COMMENT ='权限表'; - --- ---------------------------- --- Table structure for source --- ---------------------------- -DROP TABLE IF EXISTS `source`; -CREATE TABLE `source` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `config` text NOT NULL, - `type` varchar(10) NOT NULL, - `project_id` bigint(20) NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - `parent_id` bigint(20) DEFAULT NULL, - `full_parent_id` varchar(255) DEFAULT NULL, - `is_folder` tinyint(1) DEFAULT NULL, - `index` int(5) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_project_id` (`project_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for star --- ---------------------------- -DROP TABLE IF EXISTS `star`; -CREATE TABLE `star` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `target` varchar(20) NOT NULL, - `target_id` bigint(20) NOT NULL, - `user_id` bigint(20) NOT NULL, - `star_time` datetime NOT NULL ON UPDATE CURRENT_TIMESTAMP, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_target_id` (`target_id`) USING BTREE, - KEY `idx_user_id` (`user_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for user --- ---------------------------- -DROP TABLE IF EXISTS `user`; -CREATE TABLE `user` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `email` varchar(255) NOT NULL, - `username` varchar(255) NOT NULL, - `password` varchar(255) NOT NULL, - `admin` tinyint(1) NOT NULL, - `active` tinyint(1) DEFAULT NULL, - `name` varchar(255) DEFAULT NULL, - `description` varchar(255) DEFAULT NULL, - `department` varchar(255) DEFAULT NULL, - `avatar` varchar(255) DEFAULT NULL, - `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, - `create_by` bigint(20) NOT NULL DEFAULT '0', - `update_time` timestamp NOT NULL DEFAULT '1970-01-01 08:00:01', - `update_by` bigint(20) NOT NULL DEFAULT '0', - PRIMARY KEY (`id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for view --- ---------------------------- -DROP TABLE IF EXISTS `view`; -CREATE TABLE `view` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `project_id` bigint(20) NOT NULL, - `source_id` bigint(20) NOT NULL, - `sql` text, - `model` text, - `variable` text, - `config` text, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - `parent_id` bigint(20) DEFAULT NULL, - `full_parent_id` varchar(255) DEFAULT NULL, - `is_folder` tinyint(1) DEFAULT NULL, - `index` int(5) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_project_id` (`project_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - --- ---------------------------- --- Table structure for widget --- ---------------------------- -DROP TABLE IF EXISTS `widget`; -CREATE TABLE `widget` -( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(255) NOT NULL, - `description` varchar(255) DEFAULT NULL, - `view_id` bigint(20) NOT NULL, - `project_id` bigint(20) NOT NULL, - `type` bigint(20) NOT NULL, - `publish` tinyint(1) NOT NULL, - `config` longtext NOT NULL, - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - `parent_id` bigint(20) DEFAULT NULL, - `full_parent_id` varchar(255) DEFAULT NULL, - `is_folder` tinyint(1) DEFAULT NULL, - `index` int(5) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - KEY `idx_project_id` (`project_id`) USING BTREE, - KEY `idx_view_id` (`view_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8; - - -DROP TABLE IF EXISTS `rel_role_display_slide_widget`; -CREATE TABLE `rel_role_display_slide_widget` -( - `role_id` bigint(20) NOT NULL, - `mem_display_slide_widget_id` bigint(20) NOT NULL, - `visible` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`role_id`, `mem_display_slide_widget_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - - -DROP TABLE IF EXISTS `rel_role_dashboard_widget`; -CREATE TABLE `rel_role_dashboard_widget` -( - `role_id` bigint(20) NOT NULL, - `mem_dashboard_widget_id` bigint(20) NOT NULL, - `visible` tinyint(1) NOT NULL DEFAULT '0', - `create_by` bigint(20) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - PRIMARY KEY (`role_id`, `mem_dashboard_widget_id`) USING BTREE -) ENGINE = InnoDB - DEFAULT CHARSET = utf8mb4; - -DROP TABLE IF EXISTS `davinci_statistic_visitor_operation`; -CREATE TABLE `davinci_statistic_visitor_operation` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `user_id` bigint(20) DEFAULT NULL, - `email` varchar(255) DEFAULT NULL, - `action` varchar(255) DEFAULT NULL COMMENT 'login/visit/initial/sync/search/linkage/drill/download/print', - `org_id` bigint(20) DEFAULT NULL, - `project_id` bigint(20) DEFAULT NULL, - `project_name` varchar(255) DEFAULT NULL, - `viz_type` varchar(255) DEFAULT NULL COMMENT 'dashboard/display', - `viz_id` bigint(20) DEFAULT NULL, - `viz_name` varchar(255) DEFAULT NULL, - `sub_viz_id` bigint(20) DEFAULT NULL, - `sub_viz_name` varchar(255) DEFAULT NULL, - `widget_id` bigint(20) DEFAULT NULL, - `widget_name` varchar(255) DEFAULT NULL, - `variables` varchar(500) DEFAULT NULL, - `filters` varchar(500) DEFAULT NULL, - `groups` varchar(500) DEFAULT NULL, - `create_time` timestamp NULL DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8; - -DROP TABLE IF EXISTS `davinci_statistic_terminal`; -CREATE TABLE `davinci_statistic_terminal` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `user_id` bigint(20) DEFAULT NULL, - `email` varchar(255) DEFAULT NULL, - `browser_name` varchar(255) DEFAULT NULL, - `browser_version` varchar(255) DEFAULT NULL, - `engine_name` varchar(255) DEFAULT NULL, - `engine_version` varchar(255) DEFAULT NULL, - `os_name` varchar(255) DEFAULT NULL, - `os_version` varchar(255) DEFAULT NULL, - `device_model` varchar(255) DEFAULT NULL, - `device_type` varchar(255) DEFAULT NULL, - `device_vendor` varchar(255) DEFAULT NULL, - `cpu_architecture` varchar(255) DEFAULT NULL, - `create_time` timestamp NULL DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8; - - -DROP TABLE IF EXISTS `davinci_statistic_duration`; -CREATE TABLE `davinci_statistic_duration` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `user_id` bigint(20) DEFAULT NULL, - `email` varchar(255) DEFAULT NULL, - `start_time` timestamp NULL DEFAULT NULL, - `end_time` timestamp NULL DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8; - -DROP TABLE IF EXISTS `share_download_record`; -CREATE TABLE `share_download_record` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `uuid` varchar(50) DEFAULT NULL, - `name` varchar(255) NOT NULL, - `path` varchar(255) DEFAULT NULL, - `status` smallint(1) NOT NULL, - `create_time` datetime NOT NULL, - `last_download_time` datetime DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; - - -SET FOREIGN_KEY_CHECKS = 1; -INSERT INTO `source` (id,name,description,config,type,project_id,create_by,create_time,update_by,update_time,parent_id,full_parent_id,is_folder,`index`) VALUES (1,'hiveDataSource','','{"parameters":"","password":"","url":"test","username":"hiveDataSource-token"}','hive',-1,null,null,null,null,null,null,null,null); \ No newline at end of file diff --git a/db/dss_ddl.sql b/db/dss_ddl.sql index cdaf8fb1a..afa01040b 100644 --- a/db/dss_ddl.sql +++ b/db/dss_ddl.sql @@ -1,8 +1,150 @@ -SET FOREIGN_KEY_CHECKS=0; --- ---------------------------- --- Table structure for dss_application --- ---------------------------- +DROP TABLE IF EXISTS `dss_apiservice_api`; +CREATE TABLE `dss_apiservice_api` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `name` varchar(180) NOT NULL COMMENT '服务名称', + `alias_name` varchar(200) NOT NULL COMMENT '服务中文名称', + `path` varchar(180) NOT NULL COMMENT '服务路径', + `protocol` int(11) NOT NULL COMMENT '协议: http, https', + `method` varchar(10) NOT NULL COMMENT '方法： post, put, delete', + `tag` varchar(200) DEFAULT NULL COMMENT '标签', + `scope` varchar(50) DEFAULT NULL COMMENT '范围', + `description` varchar(200) DEFAULT NULL COMMENT '服务描述', + `status` int(11) DEFAULT '0' COMMENT '服务状态，默认0是停止，1是运行中，2是删除', + `type` varchar(50) DEFAULT NULL COMMENT '服务引擎类型', + `run_type` varchar(50) DEFAULT NULL COMMENT '脚本类型', + `create_time` timestamp DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `modify_time` timestamp DEFAULT CURRENT_TIMESTAMP COMMENT '修改时间', + `creator` varchar(50) DEFAULT NULL COMMENT '创建者', + `modifier` varchar(50) DEFAULT NULL COMMENT '修改者', + `script_path` varchar(180) NOT NULL COMMENT '脚本路径', + `workspaceID` int(11) NOT NULL COMMENT '工作空间ID', + `api_comment` varchar(1024) DEFAULT NULL COMMENT '服务备注', + PRIMARY KEY (`id`), + UNIQUE KEY `idx_uniq_config_name` (`name`), + UNIQUE KEY `idx_uniq_dconfig_path` (`path`), + KEY `idx_dss_script_path` (`script_path`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='服务api配置表'; + +DROP TABLE IF EXISTS `dss_apiservice_param`; +CREATE TABLE `dss_apiservice_param` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_version_id` bigint(20) NOT NULL COMMENT '服务api版本id', + `name` varchar(200) NOT NULL COMMENT '名称', + `display_name` varchar(50) DEFAULT NULL COMMENT '展示名', + `type` varchar(50) DEFAULT NULL COMMENT '类型', + `required` tinyint(1) DEFAULT '1' COMMENT '是否必须: 0否, 1是', + `default_value` varchar(1024) DEFAULT NULL COMMENT '参数的默认值', + `description` varchar(200) DEFAULT NULL COMMENT '描述', + `details` varchar(1024) DEFAULT NULL COMMENT '变量的详细说明', + PRIMARY KEY (`id`), + KEY `idx_api_version_id` (`api_version_id`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='apiservice 参数表'; + +DROP TABLE IF EXISTS `dss_apiservice_api_version`; +CREATE TABLE `dss_apiservice_api_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_id` bigint(20) NOT NULL COMMENT '服务的ID', + `version` varchar(50) NOT NULL COMMENT '服务对应的版本信息', + `bml_resource_id` varchar(50) NOT NULL COMMENT 'bml资源id', + `bml_version` varchar(20) NOT NULL COMMENT 'bml版本', + `source` varchar(200) DEFAULT NULL COMMENT '来源', + `creator` varchar(50) DEFAULT NULL COMMENT '创建者', + `create_time`timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `status` tinyint(1) default '1' COMMENT '0表示被禁用，1表示正在运行', + `metadata_info` varchar(5000) NOT NULL COMMENT '发布者库表信息', + `auth_id` varchar(200) NOT NULL COMMENT '用于与datamap交互的UUID', + `datamap_order_no` varchar(200) DEFAULT NULL COMMENT 'datamap审批单号码', + PRIMARY KEY(`id`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='服务api版本表'; + +DROP TABLE IF EXISTS `dss_apiservice_token_manager`; +CREATE TABLE `dss_apiservice_token_manager` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_version_id` bigint(20) NOT NULL COMMENT '服务api版本id', + `api_id` bigint(20) NOT NULL COMMENT '服务api配置id', + `publisher` varchar(20) NOT NULL COMMENT '发布用户', + `user` varchar(20) NOT NULL COMMENT '申请用户', + `apply_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '申请时间', + `duration` int(10) NOT NULL COMMENT '时长', + `reason` varchar(200) DEFAULT NULL COMMENT '申请原因', + `ip_whitelist` varchar(200) DEFAULT NULL COMMENT 'IP白名单', + `status` tinyint(1) DEFAULT '1' COMMENT '状态0过期，1有效期内', + `caller` varchar(50) DEFAULT NULL COMMENT '调用方', + `access_limit` varchar(50) DEFAULT NULL COMMENT '限流情况', + `apply_source` varchar(200) DEFAULT NULL COMMENT '申请来源', + `token` varchar(500) DEFAULT NULL COMMENT 'token内容', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='apiservice token管理表'; + +DROP TABLE IF EXISTS `dss_apiservice_approval`; +CREATE TABLE `dss_apiservice_approval` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_id` bigint(20) NOT NULL COMMENT '服务api id', + `api_version_id` bigint(20) NOT NULL COMMENT '版本id', + `approval_name` varchar(50) NOT NULL COMMENT '审批单名称', + `apply_user` varchar(1024) NOT NULL COMMENT '申请用户', + `execute_user` varchar(50) DEFAULT NULL COMMENT '代理执行用户，用,分割', + `creator` varchar(50) NOT NULL COMMENT '创建者', + `status` int(10) DEFAULT '0' COMMENT '申请状态，提单成功1，审批中2，成功3，失败4', + `create_time` timestamp NOT null DEFAULT CURRENT_TIMESTAMP COMMENT '审批单创建时间', + `update_time` timestamp NOT null DEFAULT CURRENT_TIMESTAMP COMMENT '审批单状态更新时间', + `approval_no` varchar(500) NOT NULL COMMENT '审批单号', + PRIMARY KEY(`id`), + UNIQUE KEY `idx_uniq_api_version_id` (`api_version_id`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='apiservice 审批单表'; + +DROP TABLE IF EXISTS `dss_apiservice_access_info`; +CREATE TABLE `dss_apiservice_access_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_id` bigint(20) NOT NULL COMMENT '服务id', + `api_version_id` bigint(20) NOT NULL COMMENT '版本id', + `api_name` varchar(50) NOT NULL COMMENT '服务名称', + `login_user` varchar(50) NOT NULL COMMENT '提交用户', + `execute_user` varchar(50) DEFAULT NULL COMMENT '代理执行用户', + `api_publisher` varchar(50) NOT NULL COMMENT 'api创建者', + `access_time` timestamp NOT null DEFAULT CURRENT_TIMESTAMP COMMENT '访问时间', + PRIMARY KEY(`id`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='apiservice 访问信息表'; + + +DROP TABLE IF EXISTS `dss_appconn`; +CREATE TABLE `dss_appconn` ( + `id` int(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `appconn_name` varchar(64) UNIQUE NOT NULL COMMENT 'appconn的名称', + `is_user_need_init` tinyint(1) DEFAULT NULL COMMENT '是否需要初始化', + `level` int(8) DEFAULT NULL COMMENT '等级', + `if_iframe` tinyint(1) DEFAULT NULL COMMENT '是否能iframe嵌入', + `is_external` tinyint(1) DEFAULT NULL COMMENT '是否是外部接入应用', + `reference` varchar(255) DEFAULT NULL COMMENT '需要关联的某一个AppConn标识', + `class_name` varchar(255) DEFAULT NULL COMMENT '需要关联的某一个AppConn标识', + `appconn_class_path` varchar(255) DEFAULT NULL COMMENT '需要关联的某一个AppConn标识', + `resource` varchar(255) DEFAULT NULL COMMENT 'bml的资源ID', + PRIMARY KEY (`id`), + UNIQUE KEY `idx_appconn_name` (`appconn_name`) +) ENGINE=InnoDB AUTO_INCREMENT=8 DEFAULT CHARSET=utf8mb4 COMMENT='dss appconn表'; + +DROP TABLE IF EXISTS `dss_appconn_instance`; +CREATE TABLE `dss_appconn_instance` ( + `id` int(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `appconn_id` int(20) NOT NULL COMMENT 'appconn的主键', + `label` varchar(128) NOT NULL COMMENT '实例的标签', + `url` varchar(128) DEFAULT NULL COMMENT '访问第三方的url', + `enhance_json` varchar(1024) DEFAULT NULL COMMENT 'json格式的配置', + `homepage_url` varchar(255) DEFAULT NULL COMMENT '主页url', + `redirect_url` varchar(255) DEFAULT NULL COMMENT '重定向url', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COMMENT='dss instance的实例表'; + +DROP TABLE IF EXISTS `dss_appconn_project_relation`; +CREATE TABLE `dss_appconn_project_relation` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `project_id` bigint(20) NOT NULL, + `appconn_instance_id` bigint(20) NOT NULL, + `appconn_instance_project_id` bigint(20) NOT NULL, + PRIMARY KEY (`id`) USING BTREE +) ENGINE=InnoDB AUTO_INCREMENT=90 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; + DROP TABLE IF EXISTS `dss_application`; CREATE TABLE `dss_application` ( `id` int(20) NOT NULL AUTO_INCREMENT, @@ -18,65 +160,186 @@ CREATE TABLE `dss_application` ( `homepage_url` varchar(255) DEFAULT NULL, `redirect_url` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; - +) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_application_user_init_result --- ---------------------------- DROP TABLE IF EXISTS `dss_application_user_init_result`; CREATE TABLE `dss_application_user_init_result` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `application_id` int(11) DEFAULT NULL, `result` varchar(255) DEFAULT NULL, - `user_id` bigint(20) DEFAULT NULL, + `username` varchar(32) DEFAULT NULL, `is_init_success` tinyint(1) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; - --- ---------------------------- --- Table structure for dss_workflow_node --- ---------------------------- -DROP TABLE IF EXISTS `dss_workflow_node`; -CREATE TABLE `dss_workflow_node` ( - `id` int(11) NOT NULL AUTO_INCREMENT, - `icon` text, - `node_type` varchar(255) DEFAULT NULL, - `application_id` int(20) DEFAULT NULL, - `submit_to_scheduler` tinyint(1) DEFAULT NULL, - `enable_copy` tinyint(1) DEFAULT NULL, - `should_creation_before_node` tinyint(1) DEFAULT NULL, - `support_jump` tinyint(1) DEFAULT NULL, - `jump_url` varchar(255) DEFAULT NULL, +DROP TABLE IF EXISTS `dss_component_info`; +CREATE TABLE `dss_component_info` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `title` varchar(128) NOT NULL, + `icon` varchar(64) NOT NULL, + `desc` varchar(1024) NOT NULL, + `button_text` varchar(64) NOT NULL, + `menu_id` int(10) NOT NULL, + `application_id` int(10) DEFAULT '0', + `user_manual_url` varchar(512) DEFAULT NULL, + `indicator_url` varchar(512) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_flow --- ---------------------------- -DROP TABLE IF EXISTS `dss_flow`; -CREATE TABLE `dss_flow` ( +DROP TABLE IF EXISTS `dss_component_role`; +CREATE TABLE `dss_component_role` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(128) DEFAULT NULL, - `state` tinyint(1) DEFAULT NULL, - `source` varchar(255) DEFAULT NULL, - `description` varchar(255) DEFAULT NULL, + `workspace_id` bigint(20) DEFAULT NULL, + `component_id` int(20) DEFAULT NULL, + `role_id` int(20) DEFAULT NULL, + `priv` int(20) DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + `updateby` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=5103 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_datawrangler_export`; +CREATE TABLE `dss_datawrangler_export` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `spread_sheet_id` int(20) NOT NULL, + `sheet_id` int(20) DEFAULT NULL, + `data_sink` varchar(1000) DEFAULT NULL, + `user_id` varchar(120) DEFAULT '0', `create_time` datetime DEFAULT NULL, - `creator_id` bigint(20) DEFAULT NULL, - `is_root_flow` tinyint(1) DEFAULT NULL, - `rank` int(10) DEFAULT NULL, - `project_id` bigint(20) DEFAULT NULL, - `has_saved` tinyint(1) DEFAULT NULL, - `uses` varchar(255) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `name` (`name`,`project_id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; + `last_update_time` datetime DEFAULT NULL, + `status` varchar(10) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_datawrangler_sheet`; +CREATE TABLE `dss_datawrangler_sheet` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(120) NOT NULL, + `spread_sheet_id` int(20) NOT NULL, + `order` int(3) DEFAULT NULL, + `data_source` varchar(1000) DEFAULT '0', + `content_location` varchar(500) DEFAULT NULL, + `operation_location` varchar(500) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `config` text, + `is_limited` tinyint(1) DEFAULT '1', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_datawrangler_spreadsheet`; +CREATE TABLE `dss_datawrangler_spreadsheet` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(120) NOT NULL, + `source` varchar(128) DEFAULT NULL, + `workspace` varchar(120) DEFAULT NULL, + `is_hidden` tinyint(1) DEFAULT '0', + `config` text, + `description` varchar(500) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_access_time` datetime DEFAULT NULL, + `access_num` int(10) DEFAULT NULL, + `user_id` varchar(120) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_datawrangler_template`; +CREATE TABLE `dss_datawrangler_template` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(120) NOT NULL, + `sheet_id` int(20) NOT NULL, + `source` varchar(120) DEFAULT NULL, + `workspace` varchar(120) DEFAULT '0', + `operation_location` varchar(500) DEFAULT NULL, + `description` varchar(500) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `user_id` varchar(120) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_dev_flow`; +CREATE TABLE `dss_dev_flow` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键ID，默认为0，所有空间都有', + `workspace_id` int(11) DEFAULT '0' COMMENT '空间ID，默认为0，所有空间都有', + `type` int(1) DEFAULT '0' COMMENT '类型: 0-空间开发流程，1-工程开发流程，2-工程编排模式', + `dev_name` varchar(200) DEFAULT NULL COMMENT '名称', + `dev_code` varchar(200) NOT NULL COMMENT '编码，可以当做checkbox或radio中的value来使用,赋值可以当做英文名称来使用', + `title` varchar(200) DEFAULT NULL COMMENT '标题', + `url` varchar(200) DEFAULT NULL COMMENT 'url', + `url_type` int(1) DEFAULT '0' COMMENT 'url类型: 0-内部系统，1-外部系统；默认是内部', + `icon` varchar(200) DEFAULT NULL COMMENT '图标', + `dev_desc` varchar(500) DEFAULT NULL COMMENT '描述', + `order_num` int(2) DEFAULT '1' COMMENT '序号', + `remark` varchar(200) DEFAULT NULL COMMENT '备注', + `create_user` varchar(100) DEFAULT NULL COMMENT '创建人', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_user` varchar(100) DEFAULT NULL COMMENT '更新人', + `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + PRIMARY KEY (`id`), + UNIQUE KEY `idx_unique_workspace_id` (`workspace_id`,`type`,`dev_code`) +) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8 COMMENT='开发流程/编排模式等配置表'; + +DROP TABLE IF EXISTS `dss_dictionary`; +CREATE TABLE `dss_dictionary` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键ID', + `workspace_id` int(11) DEFAULT '0' COMMENT '空间ID，默认为0，所有空间都有', + `parent_key` varchar(200) DEFAULT '0' COMMENT '父key', + `dic_name` varchar(200) NOT NULL COMMENT '名称', + `dic_name_en` varchar(300) DEFAULT NULL COMMENT '名称（英文）', + `dic_key` varchar(200) NOT NULL COMMENT 'key 相当于编码，空间是w_开头，工程是p_', + `dic_value` varchar(500) DEFAULT NULL COMMENT 'key对应的值', + `dic_value_en` varchar(1000) DEFAULT NULL COMMENT 'key对应的值（英文）', + `title` varchar(200) DEFAULT NULL COMMENT '标题', + `title_en` varchar(400) DEFAULT NULL COMMENT '标题（英文）', + `url` varchar(200) DEFAULT NULL COMMENT 'url', + `url_type` int(1) DEFAULT '0' COMMENT 'url类型: 0-内部系统，1-外部系统；默认是内部', + `icon` varchar(200) DEFAULT NULL COMMENT '图标', + `order_num` int(2) DEFAULT '1' COMMENT '序号', + `remark` varchar(1000) DEFAULT NULL COMMENT '备注', + `create_user` varchar(100) DEFAULT NULL COMMENT '创建人', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_user` varchar(100) DEFAULT NULL COMMENT '更新人', + `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + PRIMARY KEY (`id`), + UNIQUE KEY `idx_unique_workspace_id` (`workspace_id`,`dic_key`), + KEY `idx_parent_key` (`parent_key`), + KEY `idx_dic_key` (`dic_key`) +) ENGINE=InnoDB AUTO_INCREMENT=24 DEFAULT CHARSET=utf8 COMMENT='数据字典表'; + +DROP TABLE IF EXISTS `dss_event_relation`; +CREATE TABLE `dss_event_relation` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `project_version_id` bigint(20) NOT NULL, + `flow_id` bigint(20) NOT NULL, + `msg_type` varchar(45) NOT NULL, + `msg_topic` varchar(45) NOT NULL, + `msg_name` varchar(45) NOT NULL, + `msg_sender` varchar(45) DEFAULT NULL, + `msg_receiver` varchar(45) DEFAULT NULL, + `node_json` varchar(4096) DEFAULT NULL, + `project_id` bigint(20) NOT NULL DEFAULT '0', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='save eventchecker info for application map'; + +DROP TABLE IF EXISTS `dss_flow_edit_lock`; +CREATE TABLE `dss_flow_edit_lock` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `flow_id` bigint(11) NOT NULL, + `flow_version` varchar(16) NOT NULL, + `project_version_id` bigint(11) NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `owner` varchar(128) NOT NULL, + `lock_stamp` int(8) NOT NULL DEFAULT '0', + `is_expire` tinyint(1) NOT NULL DEFAULT '0', + `lock_content` varchar(512) NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `flow_lock` (`flow_id`,`flow_version`,`project_version_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_flow_publish_history --- ---------------------------- DROP TABLE IF EXISTS `dss_flow_publish_history`; CREATE TABLE `dss_flow_publish_history` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, @@ -90,76 +353,299 @@ CREATE TABLE `dss_flow_publish_history` ( PRIMARY KEY (`id`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_flow_relation --- ---------------------------- DROP TABLE IF EXISTS `dss_flow_relation`; CREATE TABLE `dss_flow_relation` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `flow_id` bigint(20) DEFAULT NULL, `parent_flow_id` bigint(20) DEFAULT NULL, PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +) ENGINE=InnoDB AUTO_INCREMENT=78 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_flow_taxonomy --- ---------------------------- -DROP TABLE IF EXISTS `dss_flow_taxonomy`; -CREATE TABLE `dss_flow_taxonomy` ( +DROP TABLE IF EXISTS `dss_flow_schedule_info`; +CREATE TABLE `dss_flow_schedule_info` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, - `name` varchar(20) DEFAULT NULL, - `description` varchar(255) DEFAULT NULL, - `creator_id` int(11) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `update_time` datetime DEFAULT NULL, - `project_id` bigint(20) DEFAULT NULL, - PRIMARY KEY (`id`) USING BTREE, - UNIQUE KEY `name` (`name`,`project_id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; - --- ---------------------------- --- Table structure for dss_flow_taxonomy_relation --- ---------------------------- -DROP TABLE IF EXISTS `dss_flow_taxonomy_relation`; -CREATE TABLE `dss_flow_taxonomy_relation` ( - `taxonomy_id` bigint(20) NOT NULL, - `flow_id` bigint(20) NOT NULL -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; + `flow_id` bigint(20) NOT NULL, + `project_id` bigint(20) NOT NULL, + `workspace_id` bigint(20) NOT NULL, + `schedule_time` varchar(4096) COLLATE utf8_bin DEFAULT NULL, + `alarm_level` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `alarm_user_emails` varchar(4096) COLLATE utf8_bin DEFAULT NULL, + `last_update_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; +DROP TABLE IF EXISTS `dss_flow_user`; +CREATE TABLE `dss_flow_user` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `flow_id` bigint(20) NOT NULL, + `project_id` bigint(20) NOT NULL, + `username` varchar(100) COLLATE utf8_bin NOT NULL, + `workspace_id` bigint(20) NOT NULL, + `priv` tinyint(5) NOT NULL DEFAULT '0', + `last_update_time` datetime DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_flow_version --- ---------------------------- DROP TABLE IF EXISTS `dss_flow_version`; CREATE TABLE `dss_flow_version` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `flow_id` bigint(20) DEFAULT NULL, `source` varchar(255) DEFAULT NULL, `version` varchar(255) DEFAULT NULL, + `bml_version` varchar(255) DEFAULT NULL, `json_path` text, `comment` varchar(255) DEFAULT NULL, `update_time` datetime DEFAULT NULL, - `updator_id` bigint(255) DEFAULT NULL, + `updator` varchar(32) DEFAULT NULL, `project_version_id` bigint(20) DEFAULT NULL, PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +) ENGINE=InnoDB AUTO_INCREMENT=2668 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +DROP TABLE IF EXISTS `dss_homepage_demo_instance`; +CREATE TABLE `dss_homepage_demo_instance` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `menu_id` int(20) DEFAULT NULL, + `name` varchar(64) DEFAULT NULL, + `url` varchar(128) DEFAULT NULL, + `title_en` varchar(64) DEFAULT NULL, + `title_cn` varchar(64) DEFAULT NULL, + `description` varchar(255) DEFAULT NULL, + `is_active` tinyint(1) DEFAULT '1', + `icon` varchar(255) DEFAULT NULL, + `order` int(2) DEFAULT NULL, + `click_num` int(11) DEFAULT '0', + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_homepage_demo_menu`; +CREATE TABLE `dss_homepage_demo_menu` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(64) DEFAULT NULL, + `title_en` varchar(64) DEFAULT NULL, + `title_cn` varchar(64) DEFAULT NULL, + `description` varchar(255) DEFAULT NULL, + `is_active` tinyint(1) DEFAULT '1', + `icon` varchar(255) DEFAULT NULL, + `order` int(2) DEFAULT NULL, + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_homepage_video`; +CREATE TABLE `dss_homepage_video` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(64) DEFAULT NULL, + `url` varchar(128) DEFAULT NULL, + `title_en` varchar(64) DEFAULT NULL, + `title_cn` varchar(64) DEFAULT NULL, + `description` varchar(255) DEFAULT NULL, + `is_active` tinyint(1) DEFAULT '1', + `icon` varchar(255) DEFAULT NULL, + `order` int(2) DEFAULT NULL, + `play_num` int(11) DEFAULT '0', + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_input_relation`; +CREATE TABLE `dss_input_relation` ( + `id` bigint(11) NOT NULL AUTO_INCREMENT, + `type` varchar(16) DEFAULT NULL, + `source_env` varchar(16) DEFAULT NULL, + `source_id` bigint(20) DEFAULT NULL, + `target_env` varchar(16) DEFAULT NULL, + `target_id` bigint(20) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_menu`; +CREATE TABLE `dss_menu` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(255) DEFAULT NULL, + `level` varchar(255) DEFAULT NULL, + `upper_menu_id` int(20) DEFAULT NULL, + `front_name` varchar(255) DEFAULT NULL, + `comment` varchar(255) DEFAULT NULL, + `description` varchar(255) DEFAULT NULL, + `is_active` tinyint(4) DEFAULT '1', + `is_component` tinyint(1) NOT NULL DEFAULT '0', + `icon` varchar(128) DEFAULT NULL, + `application_id` int(11) DEFAULT '0', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=34 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_menu_component_url`; +CREATE TABLE `dss_menu_component_url` ( + `id` int(10) NOT NULL AUTO_INCREMENT, + `menu_id` int(10) NOT NULL, + `dss_application_id` int(11) DEFAULT NULL, + `url` varchar(512) COLLATE utf8_bin NOT NULL, + `manul_url` varchar(512) COLLATE utf8_bin DEFAULT NULL, + `operation_url` varchar(512) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=17 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_menu_page_relation`; +CREATE TABLE `dss_menu_page_relation` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_menu_role`; +CREATE TABLE `dss_menu_role` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `workspace_id` int(20) DEFAULT NULL, + `menu_id` int(20) DEFAULT NULL, + `role_id` int(20) DEFAULT NULL, + `priv` int(20) DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + `updateby` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=5263 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_onestop_menu`; +CREATE TABLE `dss_onestop_menu` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `name` varchar(64) DEFAULT NULL, + `title_en` varchar(64) DEFAULT NULL, + `title_cn` varchar(64) DEFAULT NULL, + `description` varchar(255) DEFAULT NULL, + `is_active` tinyint(1) DEFAULT '1', + `icon` varchar(255) DEFAULT NULL, + `order` int(2) DEFAULT NULL, + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_onestop_menu_application`; +CREATE TABLE `dss_onestop_menu_application` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `application_id` int(20) DEFAULT NULL, + `onestop_menu_id` int(20) NOT NULL, + `title_en` varchar(64) DEFAULT NULL, + `title_cn` varchar(64) DEFAULT NULL, + `desc_en` varchar(255) DEFAULT NULL, + `desc_cn` varchar(255) DEFAULT NULL, + `labels_en` varchar(255) DEFAULT NULL, + `labels_cn` varchar(255) DEFAULT NULL, + `is_active` tinyint(1) DEFAULT NULL, + `access_button_en` varchar(64) DEFAULT NULL, + `access_button_cn` varchar(64) DEFAULT NULL, + `manual_button_en` varchar(64) DEFAULT NULL, + `manual_button_cn` varchar(64) DEFAULT NULL, + `manual_button_url` varchar(255) DEFAULT NULL, + `icon` varchar(255) DEFAULT NULL, + `order` int(2) DEFAULT NULL, + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + `image` varchar(200) DEFAULT NULL COMMENT '图片', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8; + + +DROP TABLE IF EXISTS `dss_onestop_user_favorites`; +CREATE TABLE `dss_onestop_user_favorites` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `username` varchar(64) DEFAULT NULL, + `workspace_id` bigint(20) DEFAULT '1', + `menu_application_id` int(20) DEFAULT NULL, + `order` int(2) DEFAULT NULL, + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=94 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_orchestrator_info`; +CREATE TABLE `dss_orchestrator_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `name` varchar(255) NOT NULL, + `type` varchar(255) NOT NULL, + `desc` varchar(1024) DEFAULT NULL, + `creator` varchar(100) NOT NULL, + `create_time` datetime DEFAULT NULL, + `project_id` bigint(20) DEFAULT NULL, + `uses` varchar(500) DEFAULT NULL, + `appconn_name` varchar(1024) NOT NULL, + `uuid` varchar(180) NOT NULL, + `secondary_type` varchar(500) DEFAULT NULL, + `is_published` tinyint(1) NOT NULL DEFAULT '0', + PRIMARY KEY (`id`) USING BTREE, + UNIQUE KEY `unique_idx_uuid` (`uuid`) + ) ENGINE=InnoDB AUTO_INCREMENT=326 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_orchestrator_schedule_info`; +CREATE TABLE `dss_orchestrator_schedule_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `orchestrator_id` bigint(20) NOT NULL, + `project_name` varchar(1024) COLLATE utf8_bin NOT NULL, + `schedule_user` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `schedule_time` varchar(4096) COLLATE utf8_bin DEFAULT NULL, + `alarm_level` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `alarm_user_emails` varchar(4096) COLLATE utf8_bin DEFAULT NULL, + `last_update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `active_flag` VARCHAR(10) DEFAULT 'true' COMMENT '调度标示：true-已启动；false-已禁用', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=39 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_orchestrator_user`; +CREATE TABLE `dss_orchestrator_user` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `orchestrator_id` bigint(20) NOT NULL, + `project_id` bigint(20) NOT NULL, + `workspace_id` int(10) NOT NULL DEFAULT '0', + `username` varchar(100) COLLATE utf8_bin NOT NULL, + `priv` tinyint(5) NOT NULL DEFAULT '0', + `last_update_time` datetime DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=58 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_orchestrator_version_info`; +CREATE TABLE `dss_orchestrator_version_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `orchestrator_id` bigint(20) NOT NULL, + `app_id` bigint(20) DEFAULT NULL, + `source` varchar(255) DEFAULT NULL, + `version` varchar(255) DEFAULT NULL, + `comment` varchar(255) DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + `updater` varchar(32) DEFAULT NULL, + `project_id` bigint(20) DEFAULT NULL, + `content` varchar(255) DEFAULT NULL, + `context_id` varchar(200) DEFAULT NULL COMMENT '上下文ID', + PRIMARY KEY (`id`) USING BTREE +) ENGINE=InnoDB AUTO_INCREMENT=422 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_project --- ---------------------------- DROP TABLE IF EXISTS `dss_project`; CREATE TABLE `dss_project` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `name` varchar(200) COLLATE utf8_bin DEFAULT NULL, `source` varchar(50) COLLATE utf8_bin DEFAULT NULL COMMENT 'Source of the dss_project', `description` text COLLATE utf8_bin, - `workspace_id` bigint(20) DEFAULT 1, `user_id` bigint(20) DEFAULT NULL, + `username` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `workspace_id` int(11) NOT NULL DEFAULT '0', `create_time` datetime DEFAULT NULL, - `create_by` bigint(20) DEFAULT NULL, + `create_by` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT '创建人', `update_time` datetime DEFAULT NULL, - `update_by` bigint(20) DEFAULT NULL, + `update_by` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT '修改人', `org_id` bigint(20) DEFAULT NULL COMMENT 'Organization ID', `visibility` bit(1) DEFAULT NULL, `is_transfer` bit(1) DEFAULT NULL COMMENT 'Reserved word', @@ -170,13 +656,15 @@ CREATE TABLE `dss_project` ( `product` varchar(200) COLLATE utf8_bin DEFAULT NULL, `application_area` tinyint(1) DEFAULT NULL, `business` varchar(200) COLLATE utf8_bin DEFAULT NULL, + `is_personal` tinyint(4) NOT NULL DEFAULT '0', + `create_by_str` varchar(256) COLLATE utf8_bin DEFAULT NULL, + `update_by_str` varchar(256) COLLATE utf8_bin DEFAULT NULL, + `dev_process` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '开发流程，多个以英文逗号分隔，取得的值是dss_dictionary中的dic_key(parent_key=p_develop_process)', + `orchestrator_mode` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '编排模式，多个以英文逗号分隔，取得的值是dss_dictionary中的dic_key(parent_key=p_arrangement_mode或下面一级)', + `visible` tinyint(4) DEFAULT '1' COMMENT '0:已删除；1：未删除(默认)', PRIMARY KEY (`id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; - +) ENGINE=InnoDB AUTO_INCREMENT=313 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_project_applications_project --- ---------------------------- DROP TABLE IF EXISTS `dss_project_applications_project`; CREATE TABLE `dss_project_applications_project` ( `project_id` bigint(20) NOT NULL, @@ -184,16 +672,33 @@ CREATE TABLE `dss_project_applications_project` ( `application_project_id` bigint(20) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; +DROP TABLE IF EXISTS `dss_project_orchestrator`; +CREATE TABLE `dss_project_orchestrator` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键ID', + `workspace_id` int(11) DEFAULT NULL COMMENT '空间id', + `project_id` int(11) DEFAULT NULL COMMENT '工程id', + `orchestrator_id` int(11) DEFAULT NULL COMMENT '编排模式id（工作流,调用orchestrator服务返回的orchestratorId）', + `orchestrator_version_id` int(11) DEFAULT NULL COMMENT '编排模式版本id（工作流,调用orchestrator服务返回的orchestratorVersionId）', + `orchestrator_name` varchar(100) DEFAULT NULL COMMENT '编排名称', + `orchestrator_mode` varchar(100) DEFAULT NULL COMMENT '编排模式，取得的值是dss_dictionary中的dic_key(parent_key=p_arrangement_mode)', + `orchestrator_way` varchar(256) DEFAULT NULL COMMENT '编排方式', + `uses` varchar(256) DEFAULT NULL COMMENT '用途', + `description` varchar(256) DEFAULT NULL COMMENT '描述', + `create_user` varchar(100) DEFAULT NULL COMMENT '创建人', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_user` varchar(100) DEFAULT NULL COMMENT '更新人', + `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + PRIMARY KEY (`id`), + KEY `idx_workspace_id` (`workspace_id`,`project_id`), + KEY `idx_orchestrator_id` (`orchestrator_id`) +) ENGINE=InnoDB AUTO_INCREMENT=197 DEFAULT CHARSET=utf8 COMMENT='DSS编排模式信息表'; --- ---------------------------- --- Table structure for dss_project_publish_history --- ---------------------------- DROP TABLE IF EXISTS `dss_project_publish_history`; CREATE TABLE `dss_project_publish_history` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `project_version_id` bigint(20) DEFAULT NULL, `create_time` datetime DEFAULT NULL, - `creator_id` bigint(20) DEFAULT NULL, + `creator` varchar(32) COLLATE utf8_bin DEFAULT NULL, `update_time` datetime DEFAULT NULL, `comment` varchar(255) COLLATE utf8_bin DEFAULT NULL, `state` tinyint(255) DEFAULT NULL, @@ -201,37 +706,38 @@ CREATE TABLE `dss_project_publish_history` ( `expire_time` datetime DEFAULT NULL, PRIMARY KEY (`id`) USING BTREE, UNIQUE KEY `project_version_id` (`project_version_id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; +) ENGINE=InnoDB AUTO_INCREMENT=102 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_project_taxonomy --- ---------------------------- DROP TABLE IF EXISTS `dss_project_taxonomy`; CREATE TABLE `dss_project_taxonomy` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `name` varchar(20) DEFAULT NULL, `description` varchar(255) DEFAULT NULL, - `creator_id` int(11) DEFAULT NULL, + `creator` varchar(32) DEFAULT NULL, `create_time` datetime DEFAULT NULL, `update_time` datetime DEFAULT NULL, PRIMARY KEY (`id`) USING BTREE, UNIQUE KEY `name` (`name`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_project_taxonomy_relation --- ---------------------------- DROP TABLE IF EXISTS `dss_project_taxonomy_relation`; CREATE TABLE `dss_project_taxonomy_relation` ( `taxonomy_id` bigint(20) NOT NULL, `project_id` bigint(20) NOT NULL, - `creator_id` bigint(11) NOT NULL + `creator` varchar(32) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; +DROP TABLE IF EXISTS `dss_project_user`; +CREATE TABLE `dss_project_user` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `project_id` int(10) NOT NULL, + `username` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `workspace_id` bigint(20) DEFAULT NULL, + `priv` int(20) DEFAULT NULL, + `last_update_time` datetime DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=1859 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; --- ---------------------------- --- Table structure for dss_project_version --- ---------------------------- DROP TABLE IF EXISTS `dss_project_version`; CREATE TABLE `dss_project_version` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, @@ -239,212 +745,386 @@ CREATE TABLE `dss_project_version` ( `version` varchar(10) COLLATE utf8_bin DEFAULT NULL, `comment` varchar(255) COLLATE utf8_bin DEFAULT NULL, `update_time` datetime DEFAULT NULL, - `updator_id` int(11) DEFAULT NULL, + `updator` varchar(32) COLLATE utf8_bin DEFAULT NULL, `lock` int(255) DEFAULT NULL, PRIMARY KEY (`id`) USING BTREE -) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; +) ENGINE=InnoDB AUTO_INCREMENT=773 DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_release_task`; +CREATE TABLE `dss_release_task` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `project_id` bigint(20) NOT NULL, + `orchestrator_version_id` bigint(20) NOT NULL, + `orchestrator_id` bigint(20) NOT NULL, + `release_user` varchar(128) NOT NULL, + `create_time` datetime DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + `status` varchar(64) DEFAULT 'init', + PRIMARY KEY (`id`) USING BTREE +) ENGINE=InnoDB AUTO_INCREMENT=605 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_role`; +CREATE TABLE `dss_role` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `workspace_id` varchar(255) DEFAULT NULL, + `name` varchar(255) DEFAULT NULL, + `front_name` varchar(255) DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + `description` varchar(512) DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `workspace_id` (`workspace_id`,`name`) +) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8 CHECKSUM=1 DELAY_KEY_WRITE=1 ROW_FORMAT=DYNAMIC; + +DROP TABLE IF EXISTS `dss_sidebar`; +CREATE TABLE `dss_sidebar` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键ID', + `workspace_id` int(11) DEFAULT '0' COMMENT '空间ID，默认为0，所有空间都有', + `name` varchar(200) DEFAULT NULL COMMENT '名称', + `name_en` varchar(400) DEFAULT NULL COMMENT '名称(英文)', + `title` varchar(200) DEFAULT NULL COMMENT '标题', + `title_en` varchar(400) DEFAULT NULL COMMENT '标题（英文）', + `type` int(1) NOT NULL COMMENT '类型: 0-知识库，1-菜单，2-常见问题', + `order_num` int(2) DEFAULT '1' COMMENT '序号，按照这个字段顺序显示', + `remark` varchar(200) DEFAULT NULL COMMENT '备注', + `create_user` varchar(100) DEFAULT NULL COMMENT '创建人', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_user` varchar(100) DEFAULT NULL COMMENT '更新人', + `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + PRIMARY KEY (`id`), + KEY `idx_workspace_id` (`workspace_id`) +) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 COMMENT='侧边栏表'; + +DROP TABLE IF EXISTS `dss_sidebar_content`; +CREATE TABLE `dss_sidebar_content` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键ID', + `workspace_id` int(11) DEFAULT '0' COMMENT '空间ID，默认为0，所有空间都有', + `sidebar_id` int(11) NOT NULL COMMENT '侧边栏ID', + `name` varchar(200) DEFAULT NULL COMMENT '名称', + `name_en` varchar(400) DEFAULT NULL COMMENT '名称（英文）', + `title` varchar(200) DEFAULT NULL COMMENT '标题', + `title_en` varchar(400) DEFAULT NULL COMMENT '标题（英文）', + `url` varchar(200) DEFAULT NULL COMMENT 'url', + `url_type` int(1) DEFAULT '0' COMMENT 'url类型: 0-内部系统，1-外部系统；默认是内部', + `icon` varchar(200) DEFAULT NULL COMMENT '图标', + `order_num` int(2) DEFAULT '1' COMMENT '序号，按照这个字段顺序显示', + `remark` varchar(200) DEFAULT NULL COMMENT '备注', + `create_user` varchar(100) DEFAULT NULL COMMENT '创建人', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_user` varchar(100) DEFAULT NULL COMMENT '更新人', + `update_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + PRIMARY KEY (`id`), + KEY `idx_sidebarws_id` (`workspace_id`,`sidebar_id`) +) ENGINE=InnoDB AUTO_INCREMENT=10 DEFAULT CHARSET=utf8 COMMENT='侧边栏-内容表'; --- ---------------------------- --- Table structure for dss_user --- ---------------------------- DROP TABLE IF EXISTS `dss_user`; CREATE TABLE `dss_user` ( - `id` int(11) NOT NULL, + `id` int(11) NOT NULL AUTO_INCREMENT, `username` varchar(64) DEFAULT NULL, `name` varchar(64) DEFAULT NULL, - `is_first_login` tinyint(1) DEFAULT NULL + `is_first_login` tinyint(1) DEFAULT NULL, + `is_admin` tinyint(1) DEFAULT '1', + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=214 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_workflow`; +CREATE TABLE `dss_workflow` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `name` varchar(128) DEFAULT NULL, + `state` tinyint(1) DEFAULT NULL, + `source` varchar(255) DEFAULT NULL, + `description` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `creator` varchar(32) DEFAULT NULL, + `is_root_flow` tinyint(1) DEFAULT NULL, + `rank` int(10) DEFAULT NULL, + `project_id` bigint(20) DEFAULT NULL, + `has_saved` tinyint(1) DEFAULT NULL, + `uses` varchar(255) DEFAULT NULL, + `bml_version` varchar(255) DEFAULT NULL, + `resource_id` varchar(255) DEFAULT NULL, + `linked_appconn_names` varchar(255) DEFAULT NULL, + `dss_labels` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`) USING BTREE +) ENGINE=InnoDB AUTO_INCREMENT=455 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT; + +DROP TABLE IF EXISTS `dss_workflow_node`; +CREATE TABLE `dss_workflow_node` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `name` varchar(16) DEFAULT NULL, + `appconn_name` varchar(64) DEFAULT '-1' COMMENT 'appconn的名称，与dss_appconn这表的appconn_name名称对应', + `node_type` varchar(255) DEFAULT NULL, + `jump_url` varchar(255) DEFAULT NULL, + `support_jump` tinyint(1) DEFAULT NULL, + `submit_to_scheduler` tinyint(1) DEFAULT NULL, + `enable_copy` tinyint(1) DEFAULT NULL, + `should_creation_before_node` tinyint(1) DEFAULT NULL, + `icon` longtext, + PRIMARY KEY (`id`) + ) ENGINE=InnoDB AUTO_INCREMENT=21 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_workflow_node_group`; +CREATE TABLE `dss_workflow_node_group` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `name` varchar(32) NOT NULL, + `name_en` varchar(32) NOT NULL, + `description` varchar(255) DEFAULT NULL, + `order` tinyint(2) NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_workflow_node_to_group`; +CREATE TABLE `dss_workflow_node_to_group` ( + `node_id` int(11) NOT NULL, + `group_id` int(11) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for event_auth --- ---------------------------- -DROP TABLE IF EXISTS `event_auth`; -CREATE TABLE `event_auth` ( - `sender` varchar(45) NOT NULL COMMENT '消息发送者', - `topic` varchar(45) NOT NULL COMMENT '消息主题', - `msg_name` varchar(45) NOT NULL COMMENT '消息名称', - `record_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '插入记录时间', - `allow_send` int(11) NOT NULL COMMENT '允许发送标志', - PRIMARY KEY (`sender`,`topic`,`msg_name`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='消息发送授权表'; +DROP TABLE IF EXISTS `dss_workflow_node_to_ui`; +CREATE TABLE `dss_workflow_node_to_ui` ( + `workflow_node_id` int(11) NOT NULL, + `ui_id` int(11) NOT NULL +) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for event_queue --- ---------------------------- -DROP TABLE IF EXISTS `event_queue`; -CREATE TABLE `event_queue` ( - `msg_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '消息ID号', - `sender` varchar(45) NOT NULL COMMENT '消息发送者', - `send_time` datetime NOT NULL COMMENT '消息发送时间', - `topic` varchar(45) NOT NULL COMMENT '消息主题', - `msg_name` varchar(45) NOT NULL COMMENT '消息名称', - `msg` varchar(250) DEFAULT NULL COMMENT '消息内容', - `send_ip` varchar(45) NOT NULL, - PRIMARY KEY (`msg_id`) -) ENGINE=InnoDB AUTO_INCREMENT=154465 DEFAULT CHARSET=utf8 COMMENT='azkaban调取系统消息队列表'; --- ---------------------------- --- Table structure for event_status --- ---------------------------- -DROP TABLE IF EXISTS `event_status`; -CREATE TABLE `event_status` ( - `receiver` varchar(45) NOT NULL COMMENT '消息接收者', - `receive_time` datetime NOT NULL COMMENT '消息接收时间', - `topic` varchar(45) NOT NULL COMMENT '消息主题', - `msg_name` varchar(45) NOT NULL COMMENT '消息名称', - `msg_id` int(11) NOT NULL COMMENT '消息的最大消费id', - PRIMARY KEY (`receiver`,`topic`,`msg_name`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='消息消费状态表'; +DROP TABLE IF EXISTS `dss_workflow_node_ui`; +CREATE TABLE `dss_workflow_node_ui` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(64) NOT NULL, + `description` varchar(255) DEFAULT NULL, + `description_en` varchar(255) DEFAULT NULL, + `lable_name` varchar(64) NOT NULL, + `lable_name_en` varchar(64) NOT NULL, + `ui_type` varchar(16) NOT NULL, + `required` tinyint(1) NOT NULL, + `value` varchar(255) DEFAULT NULL, + `default_value` varchar(255) DEFAULT NULL, + `is_hidden` tinyint(1) NOT NULL, + `condition` varchar(255) DEFAULT NULL, + `is_advanced` tinyint(1) NOT NULL, + `order` tinyint(2) NOT NULL, + `node_menu_type` tinyint(1) NOT NULL, + `is_base_info` tinyint(1) NOT NULL, + `position` varchar(32) NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=45 DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_workflow_node_ui_to_validate`; +CREATE TABLE `dss_workflow_node_ui_to_validate` ( + `ui_id` int(11) NOT NULL, + `validate_id` int(11) NOT NULL +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; + +DROP TABLE IF EXISTS `dss_workflow_node_ui_validate`; +CREATE TABLE `dss_workflow_node_ui_validate` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `validate_type` varchar(16) NOT NULL, + `validate_range` varchar(255) DEFAULT NULL, + `error_msg` varchar(255) DEFAULT NULL, + `error_msg_en` varchar(255) DEFAULT NULL, + `trigger` varchar(16) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB AUTO_INCREMENT=59 DEFAULT CHARSET=utf8mb4; + +DROP TABLE IF EXISTS `dss_workflow_project`; +CREATE TABLE `dss_workflow_project` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `name` varchar(200) COLLATE utf8_bin DEFAULT NULL, + `source` varchar(50) COLLATE utf8_bin DEFAULT NULL COMMENT 'Source of the dss_project', + `description` text COLLATE utf8_bin, + `user_id` bigint(20) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, + `create_by` bigint(20) DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + `update_by` datetime DEFAULT NULL, + `org_id` bigint(20) DEFAULT NULL COMMENT 'Organization ID', + `visibility` bit(1) DEFAULT NULL, + `is_transfer` bit(1) DEFAULT NULL COMMENT 'Reserved word', + `initial_org_id` bigint(20) DEFAULT NULL, + `isArchive` bit(1) DEFAULT b'0' COMMENT 'If it is archived', + `pic` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `star_num` int(11) DEFAULT '0', + `product` varchar(200) COLLATE utf8_bin DEFAULT NULL, + `application_area` tinyint(1) DEFAULT NULL, + `business` varchar(200) COLLATE utf8_bin DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=COMPACT; +DROP TABLE IF EXISTS `dss_workflow_project_priv`; +CREATE TABLE `dss_workflow_project_priv` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `workapce_id` bigint(20) DEFAULT NULL, + `project_id` bigint(20) DEFAULT NULL, + `user_id` bigint(20) DEFAULT NULL, + `priv` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +DROP TABLE IF EXISTS `dss_workflow_task`; +CREATE TABLE `dss_workflow_task` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `instance` varchar(50) DEFAULT NULL COMMENT 'An instance of Entrance, consists of IP address of the entrance server and port', + `exec_id` varchar(50) DEFAULT NULL COMMENT 'execution ID, consists of jobID(generated by scheduler), executeApplicationName , creator and instance', + `um_user` varchar(50) DEFAULT NULL COMMENT 'User name', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'submitUser name', + `execution_code` text COMMENT 'Run script. When exceeding 6000 lines, script would be stored in HDFS and its file path would be stored in database', + `progress` float DEFAULT NULL COMMENT 'Script execution progress, between zero and one', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the log files', + `result_location` varchar(200) DEFAULT NULL COMMENT 'File path of the result', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `created_time` datetime DEFAULT NULL COMMENT 'Creation time', + `updated_time` datetime DEFAULT NULL COMMENT 'Update time', + `run_type` varchar(50) DEFAULT NULL COMMENT 'Further refinement of execution_application_time, e.g, specifying whether to run pySpark or SparkR', + `err_code` int(11) DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `err_desc` text COMMENT 'Execution description. Generated when the execution of script fails', + `execute_application_name` varchar(200) DEFAULT NULL COMMENT 'The service a user selects, e.g, Spark, Python, R, etc', + `request_application_name` varchar(200) DEFAULT NULL COMMENT 'Parameter name for creator', + `script_path` varchar(200) DEFAULT NULL COMMENT 'Path of the script in workspace', + `params` text COMMENT 'Configuration item of the parameters', + `engine_instance` varchar(50) DEFAULT NULL COMMENT 'An instance of engine, consists of IP address of the engine server and port', + `task_resource` varchar(1024) DEFAULT NULL, + `engine_start_time` time DEFAULT NULL, + `label_json` varchar(200) DEFAULT NULL COMMENT 'label json', + PRIMARY KEY (`id`), + KEY `created_time` (`created_time`), + KEY `um_user` (`um_user`) +) ENGINE=InnoDB AUTO_INCREMENT=715 DEFAULT CHARSET=utf8mb4; --- ---------------------------- --- Table structure for dss_workspace --- ---------------------------- DROP TABLE IF EXISTS `dss_workspace`; CREATE TABLE `dss_workspace` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `name` varchar(255) DEFAULT NULL, `label` varchar(255) DEFAULT NULL, `description` varchar(255) DEFAULT NULL, + `create_by` varchar(255) DEFAULT NULL, + `create_time` datetime DEFAULT NULL, `department` varchar(255) DEFAULT NULL, `product` varchar(255) DEFAULT NULL, `source` varchar(255) DEFAULT NULL, - `create_by` varchar(255) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, + `last_update_user` varchar(30) DEFAULT NULL COMMENT '最新修改用户', PRIMARY KEY (`id`), UNIQUE KEY `name` (`name`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; +) ENGINE=InnoDB AUTO_INCREMENT=224 DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_onestop_menu --- ---------------------------- -DROP TABLE IF EXISTS `dss_onestop_menu`; -CREATE TABLE `dss_onestop_menu` ( +DROP TABLE IF EXISTS `dss_workspace_datasource`; +CREATE TABLE `dss_workspace_datasource` ( `id` int(20) NOT NULL AUTO_INCREMENT, - `name` varchar(64) DEFAULT NULL, - `title_en` varchar(64) DEFAULT NULL, - `title_cn` varchar(64) DEFAULT NULL, - `description` varchar(255) DEFAULT NULL, - `is_active` tinyint(1) DEFAULT 1, - `icon` varchar(255) DEFAULT NULL, - `order` int(2) DEFAULT NULL, - `create_by` varchar(255) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, + `workspace_id` int(20) DEFAULT NULL, + `datasource_name` varchar(255) DEFAULT NULL, + `type` varchar(255) DEFAULT NULL, + `created_time` datetime DEFAULT NULL, + `env` varchar(255) DEFAULT NULL, + `creater` varchar(255) DEFAULT NULL, + `responser` varchar(255) DEFAULT NULL, + `last_update_user` datetime DEFAULT NULL, `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_onestop_menu_application --- ---------------------------- -DROP TABLE IF EXISTS `dss_onestop_menu_application`; -CREATE TABLE `dss_onestop_menu_application` ( +DROP TABLE IF EXISTS `dss_workspace_homepage`; +CREATE TABLE `dss_workspace_homepage` ( `id` int(20) NOT NULL AUTO_INCREMENT, - `application_id` int(20) DEFAULT NULL, - `onestop_menu_id` int(20) NOT NULL, - `title_en` varchar(64) DEFAULT NULL, - `title_cn` varchar(64) DEFAULT NULL, - `desc_en` varchar(255) DEFAULT NULL, - `desc_cn` varchar(255) DEFAULT NULL, - `labels_en` varchar(255) DEFAULT NULL, - `labels_cn` varchar(255) DEFAULT NULL, - `is_active` tinyint(1) DEFAULT NULL, - `access_button_en` varchar(64) DEFAULT NULL, - `access_button_cn` varchar(64) DEFAULT NULL, - `manual_button_en` varchar(64) DEFAULT NULL, - `manual_button_cn` varchar(64) DEFAULT NULL, - `manual_button_url` varchar(255) DEFAULT NULL, - `icon` varchar(255) DEFAULT NULL, - `order` int(2) DEFAULT NULL, - `create_by` varchar(255) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, + `workspace_id` int(10) NOT NULL, + `role_id` int(20) DEFAULT NULL, + `homepage_url` varchar(256) DEFAULT NULL, + `update_time` datetime DEFAULT NULL, PRIMARY KEY (`id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; +) ENGINE=InnoDB AUTO_INCREMENT=1213 DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_onestop_user_favorites --- ---------------------------- -DROP TABLE IF EXISTS `dss_onestop_user_favorites`; -CREATE TABLE `dss_onestop_user_favorites` ( +DROP TABLE IF EXISTS `dss_workspace_public_table`; +CREATE TABLE `dss_workspace_public_table` ( `id` int(20) NOT NULL AUTO_INCREMENT, - `username` varchar(64) DEFAULT NULL, - `workspace_id` bigint(20) DEFAULT 1, - `menu_application_id` int(20) DEFAULT NULL, - `order` int(2) DEFAULT NULL, - `create_by` varchar(255) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, + `worksapce_id` int(20) DEFAULT NULL, + `table_name` varchar(255) DEFAULT NULL, + `type` varchar(255) DEFAULT NULL, + `created_time` datetime DEFAULT NULL, + `env` varchar(255) DEFAULT NULL, + `creater` varchar(255) DEFAULT NULL, + `responser` varchar(255) DEFAULT NULL, + `last_update_user` datetime DEFAULT NULL, `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_homepage_demo_menu --- ---------------------------- -DROP TABLE IF EXISTS `dss_homepage_demo_menu`; -CREATE TABLE `dss_homepage_demo_menu` ( +DROP TABLE IF EXISTS `dss_workspace_role`; +CREATE TABLE `dss_workspace_role` ( `id` int(20) NOT NULL AUTO_INCREMENT, - `name` varchar(64) DEFAULT NULL, - `title_en` varchar(64) DEFAULT NULL, - `title_cn` varchar(64) DEFAULT NULL, - `description` varchar(255) DEFAULT NULL, - `is_active` tinyint(1) DEFAULT 1, - `icon` varchar(255) DEFAULT NULL, - `order` int(2) DEFAULT NULL, - `create_by` varchar(255) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, + `workspace_id` int(20) DEFAULT NULL, + `role_id` int(20) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; --- ---------------------------- --- Table structure for dss_homepage_demo_instance --- ---------------------------- -DROP TABLE IF EXISTS `dss_homepage_demo_instance`; -CREATE TABLE `dss_homepage_demo_instance` ( - `id` int(20) NOT NULL AUTO_INCREMENT, - `menu_id` int(20) DEFAULT NULL, - `name` varchar(64) DEFAULT NULL, - `url` varchar(128) DEFAULT NULL, - `title_en` varchar(64) DEFAULT NULL, - `title_cn` varchar(64) DEFAULT NULL, - `description` varchar(255) DEFAULT NULL, - `is_active` tinyint(1) DEFAULT 1, - `icon` varchar(255) DEFAULT NULL, - `order` int(2) DEFAULT NULL, - `click_num` int(11) DEFAULT 0, - `create_by` varchar(255) DEFAULT NULL, +DROP TABLE IF EXISTS `dss_workspace_user`; +CREATE TABLE `dss_workspace_user` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `workspace_id` bigint(20) DEFAULT NULL, + `username` varchar(32) DEFAULT NULL, + `join_time` datetime DEFAULT NULL, + `created_by` varchar(255) DEFAULT NULL, + `user_id` bigint(20) DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `workspace_id` (`workspace_id`,`username`) +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 comment '空间用户表'; + +DROP TABLE IF EXISTS `dss_workspace_user_role`; +CREATE TABLE `dss_workspace_user_role` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `workspace_id` bigint(20) DEFAULT NULL, + `username` varchar(32) DEFAULT NULL, + `role_id` int(20) DEFAULT NULL, `create_time` datetime DEFAULT NULL, - `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, + `created_by` varchar(255) DEFAULT NULL, + `user_id` bigint(20) DEFAULT NULL, PRIMARY KEY (`id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; +) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 comment '空间用户角色关系表'; --- ---------------------------- --- Table structure for dss_homepage_video --- ---------------------------- -DROP TABLE IF EXISTS `dss_homepage_video`; -CREATE TABLE `dss_homepage_video` ( - `id` int(20) NOT NULL AUTO_INCREMENT, - `name` varchar(64) DEFAULT NULL, - `url` varchar(128) DEFAULT NULL, - `title_en` varchar(64) DEFAULT NULL, - `title_cn` varchar(64) DEFAULT NULL, +DROP TABLE IF EXISTS `event_auth`; +CREATE TABLE `event_auth` ( + `sender` varchar(45) NOT NULL COMMENT '消息发送者', + `topic` varchar(45) NOT NULL COMMENT '消息主题', + `msg_name` varchar(45) NOT NULL COMMENT '消息名称', + `record_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '插入记录时间', + `allow_send` int(11) NOT NULL COMMENT '允许发送标志', + PRIMARY KEY (`sender`,`topic`,`msg_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='消息发送授权表'; + +DROP TABLE IF EXISTS `event_queue`; +CREATE TABLE `event_queue` ( + `msg_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '消息ID号', + `sender` varchar(45) NOT NULL COMMENT '消息发送者', + `send_time` datetime NOT NULL COMMENT '消息发送时间', + `topic` varchar(45) NOT NULL COMMENT '消息主题', + `msg_name` varchar(45) NOT NULL COMMENT '消息名称', + `msg` varchar(250) DEFAULT NULL COMMENT '消息内容', + `send_ip` varchar(45) NOT NULL, + PRIMARY KEY (`msg_id`) +) ENGINE=InnoDB AUTO_INCREMENT=21068 DEFAULT CHARSET=utf8 COMMENT='azkaban调取系统消息队列表'; + +DROP TABLE IF EXISTS `event_status`; +CREATE TABLE `event_status` ( + `receiver` varchar(45) NOT NULL COMMENT '消息接收者', + `receive_time` datetime NOT NULL COMMENT '消息接收时间', + `topic` varchar(45) NOT NULL COMMENT '消息主题', + `msg_name` varchar(45) NOT NULL COMMENT '消息名称', + `msg_id` int(11) NOT NULL COMMENT '消息的最大消费id', + PRIMARY KEY (`receiver`,`topic`,`msg_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='消息消费状态表'; + +DROP TABLE IF EXISTS `linkis_user`; +CREATE TABLE `linkis_user` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `email` varchar(255) DEFAULT NULL, + `username` varchar(255) NOT NULL, + `password` varchar(255) DEFAULT NULL, + `admin` tinyint(1) DEFAULT NULL COMMENT 'If it is an administrator', + `active` tinyint(1) DEFAULT NULL COMMENT 'If it is active', + `name` varchar(255) DEFAULT NULL COMMENT 'User name', `description` varchar(255) DEFAULT NULL, - `is_active` tinyint(1) DEFAULT 1, - `icon` varchar(255) DEFAULT NULL, - `order` int(2) DEFAULT NULL, - `play_num` int(11) DEFAULT 0, - `create_by` varchar(255) DEFAULT NULL, - `create_time` datetime DEFAULT NULL, - `last_update_time` datetime DEFAULT NULL, - `last_update_user` varchar(30) DEFAULT NULL, + `department` varchar(255) DEFAULT NULL, + `avatar` varchar(255) DEFAULT NULL COMMENT 'Path of the avator', + `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_by` bigint(20) DEFAULT '0', + `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_by` bigint(20) DEFAULT '0', + `is_first_login` bit(1) DEFAULT NULL COMMENT 'If it is the first time to log in', PRIMARY KEY (`id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8; - +) ENGINE=InnoDB DEFAULT CHARSET=utf8; \ No newline at end of file diff --git a/db/dss_dml.sql b/db/dss_dml.sql index 78bf09bd4..3e781a62a 100644 --- a/db/dss_dml.sql +++ b/db/dss_dml.sql @@ -1,108 +1,525 @@ -INSERT INTO `dss_application` (`id`, `name`, `url`, `is_user_need_init`, `level`, `user_init_url`, `exists_project_service`, `project_url`, `enhance_json`, `if_iframe`, `homepage_url`, `redirect_url`) VALUES (NULL, 'linkis', null, '0', '1', NULL, '0', '/home', NULL, '0', '/home', NULL); -INSERT INTO `dss_application` (`id`, `name`, `url`, `is_user_need_init`, `level`, `user_init_url`, `exists_project_service`, `project_url`, `enhance_json`, `if_iframe`, `homepage_url`, `redirect_url`) VALUES (NULL, 'workflow', null, '0', '1', NULL, '0', '/workflow', NULL, '0', '/project', NULL); -INSERT INTO `dss_application` (`id`, `name`, `url`, `is_user_need_init`, `level`, `user_init_url`, `exists_project_service`, `project_url`, `enhance_json`, `if_iframe`, `homepage_url`, `redirect_url`) VALUES (NULL, 'console', null, '0', '1', NULL, '0', '/console', NULL, '0', '/console', NULL); - -SELECT @linkis_appid:=id from dss_application WHERE `name` = 'linkis'; -SELECT @workflow_appid:=id from dss_application WHERE `name` = 'workflow'; -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.python.python', @linkis_appid, '1', '1', '0', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.spark.py', @linkis_appid, '1', '1', '0', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.spark.sql', @linkis_appid, '1', '1', '0', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.spark.scala', @linkis_appid, '1', '1', '0', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.hive.hql', @linkis_appid, '1', '1', '0', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.jdbc.jdbc', @linkis_appid, '1', '1', '0', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.control.empty', @linkis_appid, '1', '1', '0', '0', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.sendemail', @linkis_appid, '1', '1', '0', '0', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.eventchecker.eventsender', @linkis_appid, '1', '1', '0', '0', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.eventchecker.eventreceiver', @linkis_appid, '1', '1', '0', '0', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.datachecker', @linkis_appid, '1', '1', '0', '0', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'workflow.subflow', @workflow_appid, '1', '0', '1', '1', NULL); - - - -INSERT INTO `dss_project_taxonomy` (`id`, `name`, `description`, `creator_id`, `create_time`, `update_time`) VALUES (NULL, 'My project', NULL, '-1', NULL, NULL); - -INSERT INTO `dss_flow_taxonomy` (`id`, `name`, `description`, `creator_id`, `create_time`, `update_time`, `project_id`) VALUES (NULL, 'My workflow', NULL, NULL, NULL,NULL, '-1'); - -UPDATE `dss_application` SET url = 'http://GATEWAY_INSTALL_IP_2:GATEWAY_PORT' WHERE `name` in('linkis','workflow'); - -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'spark.executor.instances', '取值范围：1-40，单位：个', '执行器实例最大并发数', @application_id, '2', 'NumInterval', '[1,40]', '0', '0', '2'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'spark.executor.cores', '取值范围：1-8，单位：个', '执行器核心个数', @application_id, '2', 'NumInterval', '[1,2]', '1', '0', '1'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'spark.executor.memory', '取值范围：3-15，单位：G', '执行器内存大小', @application_id, '3', 'NumInterval', '[3,15]', '0', '0', '3'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'spark.driver.cores', '取值范围：只能取1，单位：个', '驱动器核心个数', @application_id, '1', 'NumInterval', '[1,1]', '1', '1', '1'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'spark.driver.memory', '取值范围：1-15，单位：G', '驱动器内存大小', @application_id, '2', 'NumInterval', '[1,15]', '0', '0', '1'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'wds.linkis.instance', '范围：1-3，单位：个', 'spark引擎最大并发数', @application_id, '3', 'NumInterval', '[1,3]', '0', '0', '1'); - -select @key_id1:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'spark.executor.instances'; -select @key_id2:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'spark.executor.cores'; -select @key_id3:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'spark.executor.memory'; -select @key_id4:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'spark.driver.cores'; -select @key_id5:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'spark.driver.memory'; -select @key_id6:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'wds.linkis.instance'; - -SELECT @tree_id1:=t.id from linkis_config_tree t LEFT JOIN linkis_application a on t.application_id = a.id WHERE t.`name` = 'spark资源设置' and a.`name` = 'spark'; -SELECT @tree_id2:=t.id from linkis_config_tree t LEFT JOIN linkis_application a on t.application_id = a.id WHERE t.`name` = 'spark引擎设置' and a.`name` = 'spark'; - -insert into `linkis_config_key_tree` VALUES(NULL,@key_id1,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id2,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id3,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id4,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id5,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id6,@tree_id2); - -#-----------------------jdbc------------------- - -select @application_id:=id from `linkis_application` where `name` = 'nodeexecution'; -INSERT INTO `linkis_application` (`id`, `name`, `chinese_name`, `description`) SELECT NULL,'nodeexecution',`chinese_name`,`description` FROM linkis_application WHERE @application_id IS NULL LIMIT 1 ; -select @jdbc_id:=id from `linkis_application` where `name` = 'jdbc'; - -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'jdbc.url', '格式:', 'jdbc连接地址', @application_id, NULL , 'None', NULL , '0', '0', '1'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'jdbc.username', NULL , 'jdbc连接用户名', @application_id, NULL, 'None', NULL , '0', '0', '1'); -INSERT INTO `linkis_config_key` (`id`, `key`, `description`, `name`, `application_id`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`) VALUES (NULL, 'jdbc.password', NULL , 'jdbc连接密码', @application_id, NULL , 'None', NULL , '0', '0', '1'); - -select @key_id1:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'jdbc.url'; -select @key_id2:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'jdbc.username'; -select @key_id3:=id from `linkis_config_key` where `application_id` = @application_id and `key` = 'jdbc.password'; - -SELECT @tree_id1:=t.id from linkis_config_tree t LEFT JOIN linkis_application a on t.application_id = a.id WHERE t.`name` = 'jdbc连接设置' and a.`name` = 'jdbc'; - -insert into `linkis_config_key_tree` VALUES(NULL,@key_id1,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id2,@tree_id1); -insert into `linkis_config_key_tree` VALUES(NULL,@key_id3,@tree_id1); - -INSERT INTO dss_workspace (id, name, label, description, department, product, source, create_by, create_time, last_update_time, last_update_user) VALUES (1, 'default', 'default', 'default user workspace', NULL, NULL, 'create by user', 'root', NULL, NULL, 'root'); - -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 1, '工作流编辑执行', 'https://github.com/WeBankFinTech/DataSphereStudio', 'workflow edit execution', '工作流编辑执行', '工作流编辑执行', 1, NULL, 1, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 1, '工作流串联可视化', 'https://github.com/WeBankFinTech/DataSphereStudio', 'workflow series visualization', '工作流串联可视化', '工作流串联可视化', 1, NULL, 2, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 1, '工作流调度执行跑批', 'https://github.com/WeBankFinTech/DataSphereStudio', 'workflow scheduling execution run batch', '工作流调度执行跑批', '工作流调度执行跑批', 1, NULL, 3, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 2, '某业务日常运营报表', 'https://github.com/WeBankFinTech/DataSphereStudio', 'business daily operation report', '某业务日常运营报表', '某业务日常运营报表', 1, NULL, 1, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 2, '某业务机器学习建模预测', 'https://github.com/WeBankFinTech/DataSphereStudio', 'business machine learning modeling prediction', '某业务机器学习建模预测', '某业务机器学习建模预测', 1, NULL, 2, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 2, '某业务导出营销用户列表', 'https://github.com/WeBankFinTech/DataSphereStudio', 'business export marketing user list', '某业务导出营销用户列表', '某业务导出营销用户列表', 1, NULL, 3, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 3, '数据大屏体验', 'https://github.com/WeBankFinTech/DataSphereStudio', 'data big screen experience', '数据大屏体验', '数据大屏体验', 1, NULL, 1, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 3, '数据仪表盘体验', 'https://github.com/WeBankFinTech/DataSphereStudio', 'data dashboard experience', '数据仪表盘体验', '数据仪表盘体验', 1, NULL, 2, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_instance (id, menu_id, name, url, title_en, title_cn, description, is_active, icon, `order`, click_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, 3, '可视化挂件快速体验', 'https://github.com/WeBankFinTech/DataSphereStudio', 'visual widgets quick experience', '可视化挂件快速体验', '可视化挂件快速体验', 1, NULL, 3, 0, NULL, NULL, NULL, NULL); - -INSERT INTO dss_homepage_demo_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (1, 'workflow', 'workflow', '工作流', '工作流', 1, NULL, 1, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (2, 'application', 'application', '应用场景', '应用场景', 1, NULL, 2, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_demo_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (3, 'visualization', 'visualization', '可视化', '可视化', 1, NULL, 3, NULL, NULL, NULL, NULL); - -INSERT INTO dss_homepage_video (id, name, url, title_en, title_cn, description, is_active, icon, `order`, play_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, '10秒教你搭建工作流', 'https://sandbox.webank.com/wds/dss/videos/1.mp4', '10 sec how to build workflow', '10秒教你搭建工作流', '10秒教你搭建工作流', 1, NULL, 1, 0, NULL, NULL, NULL, NULL); -INSERT INTO dss_homepage_video (id, name, url, title_en, title_cn, description, is_active, icon, `order`, play_num, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, '10秒教你发邮件', 'https://sandbox.webank.com/wds/dss/videos/1.mp4', '10 sec how to send email', '10秒教你发邮件', '10秒教你发邮件', 1, NULL, 2, 0, NULL, NULL, NULL, NULL); - -INSERT INTO dss_onestop_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (1, '应用开发', 'application development', '应用开发', '应用开发描述', 1, NULL, NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (2, '数据分析', 'data analysis', '数据分析', '数据分析描述', 1, NULL, NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (3, '生产运维', 'production operation', '生产运维', '生产运维描述', 1, NULL, NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (4, '数据质量', 'data quality', '数据质量', '数据质量描述', 1, NULL, NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu (id, name, title_en, title_cn, description, is_active, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (5, '管理员功能', 'administrator function', '管理员功能', '管理员功能描述', 0, NULL, NULL, NULL, NULL, NULL, NULL); - -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 1, 'workflow development', '工作流开发', 'Workflow development is a data application development tool created by WeDataSphere with Linkis as the kernel.', '工作流开发是微众银行微数域(WeDataSphere)打造的数据应用开发工具，以任意桥(Linkis)做为内核，将满足从数据交换、脱敏清洗、分析挖掘、质量检测、可视化展现、定时调度到数据输出等数据应用开发全流程场景需求。', 'workflow, data warehouse development', '工作流,数仓开发', 1, 'enter workflow development', '进入工作流开发', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-workflow|rgb(102, 102, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 1, 'StreamSQL development', 'StreamSQL开发', 'Real-time application development is a streaming solution jointly built by WeDataSphere, Boss big data team and China Telecom ctcloud Big data team.', '实时应用开发是微众银行微数域(WeDataSphere)、Boss直聘大数据团队 和 中国电信天翼云大数据团队 社区联合共建的流式解决方案，以 Linkis 做为内核，基于 Flink Engine 构建的批流统一的 Flink SQL，助力实时化转型。', 'streaming, realtime', '流式,实时', 0, 'under union construction', '联合共建中', 'related information', '相关资讯', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-scriptis|rgb(102, 102, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 1, 'Data service development', '数据服务开发', 'Data service is a unified API service jointly built by WeDataSphere and Ihome Big data Team. With Linkis and DataSphere Studio as the kernel.', '数据服务是微众银行微数域(WeDataSphere)与 艾佳生活大数据团队 社区联合共建的统一API服务，以 Linkis 和 DataSphere Studio 做为内核，提供快速将 Scriptis 脚本生成数据API的能力，协助企业统一管理对内对外的API服务。', 'API, data service', 'API,数据服务', 0, 'under union construction', '联合共建中', 'related information', '相关资讯', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-scriptis|rgb(102, 102, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 2, 'Scriptis', 'Scriptis', 'Scriptis is a one-stop interactive data exploration analysis tool built by WeDataSphere, uses Linkis as the kernel.', 'Scriptis是微众银行微数域(WeDataSphere)打造的一站式交互式数据探索分析工具，以任意桥(Linkis)做为内核，提供多种计算存储引擎(如Spark、Hive、TiSpark等)、Hive数据库管理功能、资源(如Yarn资源、服务器资源)管理、应用管理和各种用户资源(如UDF、变量等)管理的能力。', 'scripts development,IDE', '脚本开发,IDE', 1, 'enter Scriptis', '进入Scriptis', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-scriptis|rgb(102, 102, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 2, 'Visualis', 'Visualis', 'Visualis is a data visualization BI tool based on Davinci, with Linkis as the kernel, it supports the analysis mode of data development exploration.', 'Visualis是基于宜信开源项目Davinci开发的数据可视化BI工具，以任意桥(Linkis)做为内核，支持拖拽式报表定义、图表联动、钻取、全局筛选、多维分析、实时查询等数据开发探索的分析模式，并做了水印、数据质量校验等金融级增强。', 'visualization, statement', '可视化,报表', 1, 'enter Visualis', '进入Visualis', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-visualis|rgb(0, 153, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 3, 'Schedulis', 'Schedulis', 'Description for Schedulis.', 'Schedulis描述', 'scheduling, workflow', '调度,工作流', 1, 'enter Schedulis', '进入Schedulis', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-schedule|rgb(102, 102, 204)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 3, 'Application operation center', '应用运维中心', 'Description for Application operation center.', '应用运维中心描述', 'production, operation', '生产,运维', 0, 'enter application operation center', '进入应用运维中心', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-scriptis|rgb(102, 102, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 4, 'Qualitis', 'Qualitis', 'Qualitis is a financial and one-stop data quality management platform that provides data quality model definition, visualization and monitoring of data quality results', 'Qualitis是一套金融级、一站式的数据质量管理平台，提供了数据质量模型定义，数据质量结果可视化、可监控等功能，并用一整套统一的流程来定义和检测数据集的质量并及时报告问题。', 'product, operations', '生产,运维', 1, 'enter Qualitis', '进入Qualitis', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-qualitis|rgb(51, 153, 153)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 4, 'Exchangis', 'Exchangis', 'Exchangis is a lightweight, high scalability, data exchange platform, support for structured and unstructured data transmission between heterogeneous data sources.', 'Exchangis是一个轻量级的、高扩展性的数据交换平台，支持对结构化及无结构化的异构数据源之间的数据传输，在应用层上具有数据权限管控、节点服务高可用和多租户资源隔离等业务特性，而在数据层上又具有传输架构多样化、模块插件化和组件低耦合等架构特点。', 'user manual', '生产,运维', 1, 'enter Exchangis', '进入Exchangis', 'user manual', '用户手册', 'https://github.com/WeBankFinTech/DataSphereStudio', 'fi-exchange|(102, 102, 255)', NULL, NULL, NULL, NULL, NULL); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 5, 'Workspace management', '工作空间管理', NULL, NULL, NULL, NULL, 1, 'workspace management', '工作空间管理', null, null, null, 'fi-scriptis|rgb(102, 102, 255)', null, null, null, null, null); -INSERT INTO dss_onestop_menu_application (id, application_id, onestop_menu_id, title_en, title_cn, desc_en, desc_cn, labels_en, labels_cn, is_active, access_button_en, access_button_cn, manual_button_en, manual_button_cn, manual_button_url, icon, `order`, create_by, create_time, last_update_time, last_update_user) VALUES (NULL, NULL, 5, 'User resources management', '用户资源管理', NULL, NULL, NULL, NULL, 1, 'user resource management', '用户资源管理', null, null, null, 'fi-scriptis|rgb(102, 102, 255)', null, null, null, null, null); +DELETE FROM dss_workspace; +insert into `dss_workspace` (`name`, `label`, `description`, `create_by`, `create_time`, `department`, `product`, `source`, `last_update_time`, `last_update_user`) values('bdapWorkspace','','bdapWorkspace','hadoop','2020-07-13 02:39:41','企业直通银行部','bdapWorkspace',NULL,'2020-07-13 02:39:41','hadoop'); +DELETE FROM dss_dictionary; +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (1,0,'0','空间开发流程','Space development process','w_develop_process',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'空间开发流程','SYSTEM','2020-12-28 17:32:34',NULL,'2021-02-22 17:46:40'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (2,0,'w_develop_process','需求','Demand','wdp_demand','创建新的业务需求，并将需求指派给对应负责人。','Create new business requirements and assign them to the corresponding responsible person.','Demo案例','Demo case',NULL,0,'xuqiu',1,'空间开发流程-需求','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-23 09:38:07'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (3,0,'w_develop_process','设计','Design','wdp_design','针对新的业务需求，进行数仓规划和库表设计。','According to the new business requirements, data warehouse planning and database table design are carried out.','Demo案例','Demo case',NULL,0,'sheji',1,'空间开发流程-设计','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-23 09:38:09'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (4,0,'w_develop_process','开发','Development','wdp_development','针对新的业务需求，进行数仓规划和库表设计。','According to the new business requirements, data warehouse planning and database table design are carried out.','Demo案例','Demo case',NULL,0,'kaifa',1,'空间开发流程-开发','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-23 09:38:10'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (5,0,'w_develop_process','调试','Debugging','wdp_debug','创建新的业务需求，并将需求指派给对应负责人。','Create new business requirements and assign them to the corresponding responsible person.','Demo案例','Demo case',NULL,0,'tiaoshi',1,'空间开发流程-调试','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-23 09:38:11'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (6,0,'w_develop_process','生产','Production','wdp_product','创建新的业务需求，并将需求指派给对应负责人。','Create new business requirements and assign them to the corresponding responsible person.','Demo案例','Demo case',NULL,0,'shengchan',1,'空间开发流程-生产','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-23 09:38:12'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (7,0,'0','工程开发流程','Engineering development process','p_develop_process',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程开发流程','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:48:48'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (8,0,'p_develop_process','开发中心','Development Center','pdp_development_center','dev',NULL,NULL,NULL,NULL,0,'kaifa-icon',1,'工程开发流程-开发中心','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:49:02'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (10,0,'0','工程编排模式','Project layout mode','p_orchestrator_mode',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程编排模式','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:49:36'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (11,0,'p_orchestrator_mode','工作流','Workflow','pom_work_flow','radio',NULL,NULL,NULL,NULL,0,'gongzuoliu-icon',1,'工程编排模式-工作流','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:49:49'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (14,0,'pom_work_flow','DAG','DAG','pom_work_flow_DAG',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程编排模式-工作流-DAG','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:50:31'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (16,0,'pom_single_task','Scriptis','Scriptis','pom_single_task_scriptis',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程编排模式-单任务-Scriptis','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:51:08'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (18,0,'pom_single_task','Qualitis','Qualitis','pom_single_task_qualitis',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程编排模式-单任务-Qualitis','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:50:53'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (20,0,'pom_consist_orchestrator','Qualitis','Qualitis','pom_consist_orchestrator_qualitis',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程编排模式-组合编排-Qualitis','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:50:57'); +insert into `dss_dictionary`(`id`,`workspace_id`,`parent_key`,`dic_name`,`dic_name_en`,`dic_key`,`dic_value`,`dic_value_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (21,0,'pom_consist_orchestrator','Email','Email','pom_consist_orchestrator_email',NULL,NULL,NULL,NULL,NULL,0,NULL,1,'工程编排模式-组合编排-Email','SYSTEM','2020-12-28 17:32:35',NULL,'2021-02-22 17:51:22'); +insert into `dss_dictionary` (`workspace_id`, `parent_key`, `dic_name`, `dic_name_en`, `dic_key`, `dic_value`, `dic_value_en`, `title`, `title_en`, `url`, `url_type`, `icon`, `order_num`, `remark`, `create_user`, `create_time`, `update_user`, `update_time`) values('0','0','工作空间默认部门','Space development name','w_workspace_department','10001-部门一;10002-部门二;10003-部门三',NULL,NULL,NULL,NULL,'0',NULL,'1','工作空间默认部门，前面是id后面是部门名称中间使用‘-’,横杆分隔，多个以英文分号分隔','SYSTEM','2020-12-28 17:32:34',NULL,'2021-02-22 17:46:40'); + +DELETE FROM dss_menu; +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (2,'develop_center','1',0,'开发中心',NULL,NULL,1,0,'icon-kaifazhongxinmorenzhuangtai',0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (3,'analysis_center','1',0,'分析中心',NULL,NULL,1,0,'icon-fenxizhongxin',0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (8,'schedule_center','2',1,'调度中心',NULL,NULL,1,0,NULL,0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (12,'workflow','2',2,'工作流开发',NULL,NULL,1,1,NULL,2); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (16,'scriptis','2',3,'意书(Scrptis)',NULL,NULL,1,1,NULL,1); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (28,'settings','1',0,'设置',NULL,NULL,1,0,'icon-shezhi',0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (29,'workspace_setting','2',28,'工作空间设置',NULL,NULL,1,0,NULL,0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (30,'user_manage','2',28,'用户管理',NULL,NULL,1,0,NULL,0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (31,'priv_manage','2',28,'权限管理',NULL,NULL,1,0,NULL,0); +insert into `dss_menu`(`id`,`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values (32,'apiService','2',2,'数据服务',NULL,NULL,1,1,NULL,11); + +DELETE FROM dss_role; +insert into `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) values('1','-1','admin','管理员','2020-07-13 02:43:35','通用角色管理员'); +insert into `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) values('2','-1','maintenance','运维用户','2020-07-13 02:43:35','通用角色运维用户'); +insert into `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) values('3','-1','developer','开发用户','2020-07-13 02:43:35','通用角色开发用户'); +insert into `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) values('4','-1','analyser','分析用户','2020-07-13 02:43:36','通用角色分析用户'); +insert into `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) values('5','-1','operator','运营用户','2020-07-13 02:43:36','通用角色运营用户'); +insert into `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) values('6','-1','boss','领导','2020-07-13 02:43:36','通用角色领导'); + +DELETE FROM dss_sidebar; +insert into `dss_sidebar`(`id`,`workspace_id`,`name`,`name_en`,`title`,`title_en`,`type`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (1,0,'知识库','Knowledge base','知识库','Knowledge base',0,1,NULL,'SYSTEM','2020-12-15 13:21:06',NULL,'2021-02-23 09:45:41'); +insert into `dss_sidebar`(`id`,`workspace_id`,`name`,`name_en`,`title`,`title_en`,`type`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (2,0,'菜单','Menu','菜单','Menu',1,1,NULL,'SYSTEM','2020-12-15 13:21:06',NULL,'2021-02-23 09:45:50'); +insert into `dss_sidebar`(`id`,`workspace_id`,`name`,`name_en`,`title`,`title_en`,`type`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (3,0,'常见问题','Common problem','常见问题','Common problem',1,1,NULL,'SYSTEM','2020-12-15 13:21:06',NULL,'2021-02-23 09:46:18'); + +DELETE FROM dss_sidebar_content; +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (1,0,1,NULL,NULL,'部门队列申请','Department application','http://127.0.0.1:8088/kn/d/38',1,'wendang',1,NULL,'SYSTEM','2020-12-15 13:21:06',NULL,'2021-02-23 09:47:29'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (2,0,1,NULL,NULL,'部门导数申请','Departmental derivative','http://127.0.0.1:8088/kn/d/40',1,'wendang',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-02-23 09:47:36'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (3,0,2,NULL,NULL,'工作空间管理','Workspace management','/workspaceManagement/productsettings',0,'menuIcon',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-02-23 09:47:49'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (4,0,2,NULL,NULL,'知识库','Knowledge base','http://127.0.0.1:8088/kn/d/40',1,'menuIcon',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-02-23 09:47:11'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (5,0,3,NULL,NULL,'资源配置说明',NULL,'http://127.0.0.1:8088/kn/d/38',1,'fi-warn',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-01-12 17:16:52'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (6,0,3,NULL,NULL,'Spark使用指南','[Discussion on error code 22223]','http://127.0.0.1:8088/kn/d/40',1,'fi-warn',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-02-23 09:48:28'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (7,0,3,NULL,NULL,'Hive语法介绍',NULL,'http://127.0.0.1:8088/kn/d/34',1,'fi-warn',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-01-12 17:17:00'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (8,0,3,NULL,NULL,'工作流使用介绍',NULL,'http://127.0.0.1:8088/kn/d/42',1,'fi-warn',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-01-12 17:17:01'); +insert into `dss_sidebar_content`(`id`,`workspace_id`,`sidebar_id`,`name`,`name_en`,`title`,`title_en`,`url`,`url_type`,`icon`,`order_num`,`remark`,`create_user`,`create_time`,`update_user`,`update_time`) values (9,0,3,NULL,NULL,'数据服务使用介绍','Discussion on error code 22223','http://127.0.0.1:8088/kn/d/32',1,'fi-warn',1,NULL,'SYSTEM','2020-12-15 13:21:07',NULL,'2021-02-23 09:48:19'); + +DELETE FROM dss_onestop_menu; +INSERT INTO `dss_onestop_menu` (`id`, `name`, `title_en`, `title_cn`, `description`, `is_active`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`) VALUES('1','应用开发','application development','应用开发','应用开发描述','1',NULL,NULL,NULL,NULL,NULL,NULL); +INSERT INTO `dss_onestop_menu` (`id`, `name`, `title_en`, `title_cn`, `description`, `is_active`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`) VALUES('2','数据分析','data analysis','数据分析','数据分析描述','1',NULL,NULL,NULL,NULL,NULL,NULL); +INSERT INTO `dss_onestop_menu` (`id`, `name`, `title_en`, `title_cn`, `description`, `is_active`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`) VALUES('3','生产运维','production operation','生产运维','生产运维描述','1',NULL,NULL,NULL,NULL,NULL,NULL); +INSERT INTO `dss_onestop_menu` (`id`, `name`, `title_en`, `title_cn`, `description`, `is_active`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`) VALUES('4','数据质量','data quality','数据质量','数据质量描述','1',NULL,NULL,NULL,NULL,NULL,NULL); +INSERT INTO `dss_onestop_menu` (`id`, `name`, `title_en`, `title_cn`, `description`, `is_active`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`) VALUES('5','管理员功能','administrator function','管理员功能','管理员功能描述','0',NULL,NULL,NULL,NULL,NULL,NULL); + +DELETE FROM dss_onestop_menu_application; +INSERT INTO `dss_onestop_menu_application` (`id`, `application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES('2',NULL,'1','StreamSQL development','StreamSQL开发','Real-time application development is a streaming solution jointly built by WeDataSphere, Boss big data team and China Telecom ctcloud Big data team.','实时应用开发是微众银行微数域(WeDataSphere)、Boss直聘大数据团队 和 中国电信天翼云大数据团队 社区联合共建的流式解决方案，以 Linkis 做为内核，基于 Flink Engine 构建的批流统一的 Flink SQL，助力实时化转型。','streaming, realtime','流式,实时','0','under union construction','联合共建中','related information','相关资讯','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujukaifa-logo',NULL,NULL,NULL,NULL,NULL,'shujukaifa-icon'); +INSERT INTO `dss_onestop_menu_application` (`id`, `application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES('3','11','1','Data service development','数据服务开发','Data service is a unified API service jointly built by WeDataSphere and Ihome Big data Team. With Linkis and DataSphere Studio as the kernel.','数据服务是微众银行微数域(WeDataSphere)与 艾佳生活大数据团队 社区联合共建的统一API服务，以 Linkis 和 DataSphere Studio 做为内核，提供快速将 Scriptis 脚本生成数据API的能力，协助企业统一管理对内对外的API服务。','API, data service','API,数据服务','1','under union construction','进入数据服务','related information','相关资讯','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujufuwu-logo',NULL,NULL,NULL,NULL,NULL,'shujufuwu-icon'); +INSERT INTO `dss_onestop_menu_application` (`id`, `application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES('4','1','2','Scriptis','Scriptis','Scriptis is a one-stop interactive data exploration analysis tool built by WeDataSphere, uses Linkis as the kernel.','Scriptis是微众银行微数域(WeDataSphere)打造的一站式交互式数据探索分析工具，以任意桥(Linkis)做为内核，提供多种计算存储引擎(如Spark、Hive、TiSpark等)、Hive数据库管理功能、资源(如Yarn资源、服务器资源)管理、应用管理和各种用户资源(如UDF、变量等)管理的能力。','scripts development,IDE','脚本开发,IDE','1','enter Scriptis','进入Scriptis','user manual','用户手册','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujukaifa-logo',NULL,NULL,NULL,NULL,NULL,'shujukaifa-icon'); + +INSERT INTO `dss_onestop_menu_application` (`id`, `application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES('9',NULL,'4','Exchangis','Exchangis','Exchangis is a lightweight, high scalability, data exchange platform, support for structured and unstructured data transmission between heterogeneous data sources.','Exchangis是一个轻量级的、高扩展性的数据交换平台，支持对结构化及无结构化的异构数据源之间的数据传输，在应用层上具有数据权限管控、节点服务高可用和多租户资源隔离等业务特性，而在数据层上又具有传输架构多样化、模块插件化和组件低耦合等架构特点。','user manual','生产,运维','0','enter Exchangis','进入Exchangis','user manual','用户手册','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujujiaohuan-logo',NULL,NULL,NULL,NULL,NULL,'shujujiaohuan-icon'); +INSERT INTO `dss_onestop_menu_application` (`id`, `application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES('10','7','5','Workspace management','工作空间管理',NULL,NULL,NULL,NULL,'1','workspace management','工作空间管理',NULL,NULL,NULL,'shujukaifa-logo',NULL,NULL,NULL,NULL,NULL,'shujukaifa-icon'); +INSERT INTO `dss_onestop_menu_application` (`id`, `application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES('11',NULL,'5','User resources management','用户资源管理',NULL,NULL,NULL,NULL,'1','user resource management','用户资源管理',NULL,NULL,NULL,'shujukaifa-logo',NULL,NULL,NULL,NULL,NULL,'shujukaifa-icon'); + +DELETE FROM dss_role; +DELETE FROM dss_workspace_user_role; +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('1','-1','admin','管理员','2020-07-13 02:43:35','通用角色管理员'); +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('2','-1','maintenance','运维用户','2020-07-13 02:43:35','通用角色运维用户'); +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('3','-1','developer','开发用户','2020-07-13 02:43:35','通用角色开发用户'); +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('4','-1','analyser','分析用户','2020-07-13 02:43:36','通用角色分析用户'); +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('5','-1','operator','运营用户','2020-07-13 02:43:36','通用角色运营用户'); +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('6','-1','boss','领导','2020-07-13 02:43:36','通用角色领导'); +INSERT INTO `dss_role` (`id`, `workspace_id`, `name`, `front_name`, `update_time`, `description`) VALUES('7','-1','apiUser','数据服务用户','2020-08-21 11:35:02','通用角色数据服务用户'); + +DELETE FROM dss_application; +INSERT INTO `dss_application`(`id`,`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES (1,'linkis','http://127.0.0.1:9001',0,1,NULL,0,'/home','{\"watermark\":false,\"rsDownload\":true}',0,'/home',NULL); +INSERT INTO `dss_application`(`id`,`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES (4,'workflow','http://127.0.0.1:9001',0,1,NULL,0,'/workflow',NULL,0,'/project',NULL); +INSERT INTO `dss_application`(`id`,`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES (5,'console',NULL,0,1,NULL,0,'/console',NULL,0,'/console',NULL); +INSERT INTO `dss_application`(`id`,`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES (7,'workspace management','/workspaceManagement',0,1,NULL,0,NULL,NULL,NULL,'/workspaceManagement',NULL); +INSERT INTO `dss_application`(`id`,`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES (11,'apiService','http://127.0.0.1:9001',0,1,NULL,0,'/apiservices',NULL,0,'/apiservices',NULL); + +UPDATE `dss_application` SET url = 'http://GATEWAY_INSTALL_IP:GATEWAY_PORT' WHERE `name` in('linkis','workflow'); + +DELETE FROM dss_project_taxonomy; +INSERT INTO `dss_project_taxonomy` (`id`, `name`, `description`, `creator`, `create_time`, `update_time`) VALUES (NULL, 'My project', NULL, '-1', NULL, NULL); + +DELETE FROM dss_workflow_node; +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('1','python','-1','linkis.python.python',NULL,'1','1','1','0','pythonCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('2','pyspark','-1','linkis.spark.py',NULL,'1','1','1','0','pysparkCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('3','sql','-1','linkis.spark.sql',NULL,'1','1','1','0','sqlCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('4','scala','-1','linkis.spark.scala',NULL,'1','1','1','0','ScalaCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('5','hql','-1','linkis.hive.hql',NULL,'1','1','1','0',' hqlCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('6','jdbc','-1','linkis.jdbc.jdbc',NULL,'1','1','1','0',''); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('7','connector','-1','linkis.control.empty',NULL,'0','1','1','0','connectorCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('8','sendemail','-1','linkis.appconn.sendemail',NULL,'0','1','1','0','sendemailCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('9','eventsender','-1','linkis.appconn.eventchecker.eventsender',NULL,'0','1','1','0','eventsenderCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('10','eventreceiver','-1','linkis.appconn.eventchecker.eventreceiver',NULL,'0','1','1','0','eventcheckerCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('11','datachecker','-1','linkis.appconn.datachecker',NULL,'0','1','1','0','datacheckerCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('12','subFlow','-1','workflow.subflow',NULL,'1','1','0','1','subflowCreated with Sketch.'); +insert into `dss_workflow_node` (`id`, `name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('15','shell','-1','linkis.shell.sh',NULL,'1','1','1','0','shellCreated with Sketch.'); + +DELETE FROM dss_workflow_node_group; +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (1,'数据交换','Data exchange',NULL,1); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (2,'数据开发','Data development',NULL,2); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (3,'数据质量','Data Givernance',NULL,3); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (4,'数据可视化','Data visualization',NULL,4); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (5,'数据输出','Data output',NULL,8); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (6,'信号节点','Signal node',NULL,6); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (7,'功能节点','Function node',NULL,7); +insert into `dss_workflow_node_group`(`id`,`name`,`name_en`,`description`,`order`) values (8,'机器学习','Machine Learning',NULL,5); + +DELETE FROM dss_workflow_node_to_group; +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (1,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (2,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (3,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (4,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (5,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (6,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (15,2); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (16,3); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (8,5); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (9,6); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (10,6); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (11,6); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (12,7); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (7,7); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (18,3); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (19,3); +insert into `dss_workflow_node_to_group`(`node_id`,`group_id`) values (20,8); + +DELETE FROM dss_workflow_node_to_ui; +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,7); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,8); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,9); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,10); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,11); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (3,12); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (1,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (1,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (1,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (1,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,7); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,8); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,9); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,10); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,11); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (2,12); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,7); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,8); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,9); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,10); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (4,11); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (5,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (5,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (5,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (5,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (5,11); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (7,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (7,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (7,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (7,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,13); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,14); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,15); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,16); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,17); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (8,18); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,20); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,39); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,22); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,40); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,24); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,21); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (9,23); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,25); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,26); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,27); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,28); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,29); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (10,30); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,31); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,32); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,33); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (11,34); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (12,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (12,2); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (12,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (12,4); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (12,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (12,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (13,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (13,2); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (13,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (13,4); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (13,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (13,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (14,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (14,2); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (14,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (14,4); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (14,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (14,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (15,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (15,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (15,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (15,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (16,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (16,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (16,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (16,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (16,35); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (16,36); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,5); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,6); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,37); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,2); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (17,4); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (18,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (18,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (18,41); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (18,42); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (20,43); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (20,44); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (20,1); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (20,3); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (20,2); +insert into `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (20,4); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (1,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (2,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (3,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (4,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (5,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (6,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (7,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (8,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (9,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (10,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (11,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (12,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (13,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (14,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (15,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (17,45); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) VALUES (20,45); + +DELETE FROM dss_workflow_node_ui; +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (1,'title','请填写节点名称','Please enter node name','节点名','Node name','Input',1,NULL,NULL,0,NULL,0,1,1,1,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (2,'title','请填写节点名称','Please enter node name','节点名','Node name','Input',1,NULL,NULL,0,NULL,0,1,0,1,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (3,'desc','请填写节点描述','Please enter the node description','节点描述','Node description','Text',0,NULL,NULL,0,NULL,0,4,1,1,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (4,'desc','请填写节点描述','Please enter the node description','节点描述','Node description','Text',0,NULL,NULL,0,NULL,0,2,0,1,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (5,'businessTag',NULL,NULL,'业务标签','businessTag','Tag',0,NULL,NULL,0,NULL,0,2,1,1,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (6,'appTag',NULL,NULL,'应用标签','appTag','Tag',0,NULL,NULL,0,NULL,0,3,1,1,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (7,'spark.driver.memory','驱动器内存大小，默认值：2','Driver memory, default value: 2','spark-driver-memory','spark-driver-memory','Input',0,NULL,'2',0,NULL,0,1,1,0,'startup'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (8,'spark.executor.memory','执行器内存大小，默认值：3','Executor memory, default value: 3','spark-executor-memory','spark-executor-memory','Input',0,NULL,'3',0,NULL,0,1,1,0,'startup'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (9,'spark.executor.cores','执行器核心个数，默认值：1','Number of cores per executor, default value: 1','spark-executor-cores','spark-executor-cores','Input',0,NULL,'2',0,NULL,0,1,1,0,'startup'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (10,'spark.executor.instances','执行器个数，默认值：2','Number of executors, default value: 2','spark-executor-instances','spark-executor-instances','Input',0,NULL,'2',0,NULL,0,1,1,0,'startup'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (11,'wds.linkis.rm.yarnqueue','执行队列','Execution queue','wds-linkis-yarnqueue','wds-linkis-yarnqueue','Input',0,NULL,'dws',0,NULL,0,1,1,0,'startup'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (12,'resources',NULL,NULL,'资源信息','Resource information','Upload',0,'[]',NULL,0,NULL,0,1,1,0,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (13,'category','请选择类型','Please choose the type','类型','Type','Select',1,'[\"node\"]','node',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (14,'subject','请填写邮件标题','Please enter the email subject','邮件标题','Email Subject','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (15,'content','请选择或输入发送项','Please choose or enter the items to send','发送项','Intems to Send','MultiBinding',1,'[\"linkis.appconn.visualis.display\",\"linkis.appconn.visualis.dashboard\"]','[]',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (16,'to','请填写收件人','Please enter recipients','收件人','To','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (17,'cc','请填写抄送人','Please enter carbon copy recipients','抄送','Cc','Input',0,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (18,'bcc','请填写秘密发送人','Please enter blind carbon copy recipients','秘密抄送','Bcc','Input',0,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (19,'itsm','请填写关联审批单','Please enter ITSM','关联审批单','ITSM','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (20,'msg.type','请正确填写消息类型','Please enter message type correctly','msg.type','msg.type','Disable',1,NULL,'SEND',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (21,'msg.topic','消息主题，必须与eventreceiver完全一致','Message subject must be exactly the same as eventreceiver','msg.topic','msg.topic','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (22,'msg.sender','请正确填写发送者','Please enter the sender correctly','msg.sender','msg.sender','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (23,'msg.name','消息名称，必须与eventreceiver完全一致','The message name must be exactly the same as the eventreceiver','msg.name','msg.name','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (24,'msg.body','请正确填写消息内容','Please enter the message content correctly','msg.body','msg.body','Text',0,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (25,'msg.type','请正确填写消息类型','Please enter message type correctly','msg.type','msg.type','Disable',1,NULL,'RECEIVE',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (26,'msg.receiver','请正确填写消息接收者','Please enter message recipients correctly','msg.receiver','msg.receiver','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (27,'query.frequency','请填写查询频率，默认10次','Please enter query frequency, 10 times by default','query.frequency','query.frequency','Disable',0,NULL,'10',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (28,'max.receive.hours','请填写等待时间，默认1小时','Please enter waiting time, 1 hour by default','max.receive.hours','max.receive.hours','Input',0,NULL,'12',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (29,'msg.savekey','消息共享key值，默认msg.body','The ky of message content, msg.body by default','msg.savekey','msg.savekey','Input',0,NULL,'msg.body',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (30,'only.receive.today',NULL,NULL,'only.receive.today','only.receive.today','Input',0,NULL,'true',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (31,'source.type','请选择数据来源','Please choose the data source','source.type','source.type','Select',1,'[\"hivedb\",\"maskdb\"]',NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (32,'check.object','比如：db.tb{ds=${run_date}}','Please enter the name of data dependency','check.object','check.object','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (33,'max.check.hours',NULL,NULL,'max.check.hours','max.check.hours','Input',0,NULL,'1',0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (34,'job.desc','请正确填写多源配置','Please enter multi-source configuration correctly','job.desc','job.desc','Text',0,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (35,'filter',NULL,NULL,'过滤条件','Filter','Input',0,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (36,'executeUser','必填','Required','执行用户','Executor','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (37,'bindViewKey','请选择上游节点','Please select upstream node','绑定上游节点','Bind front node','Binding',1,'[\"*\"]','empty',0,NULL,0,3,0,0,'node'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (39,'msg.topic','消息主题，必须与eventsender完全一致','Message subject must be exactly the same as eventsender','msg.topic','msg.topic','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (40,'msg.name','消息名称，必须与eventsender完全一致','The message name must be exactly the same as the eventsender ','msg.name','msg.name','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (41,'executeUser','请填写执行用户','Please enter execute user','执行用户','executeUser','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (42,'Filter','请填写过滤条件','Please enter filter','过滤条件','Filter','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (43,'executeUser','请填写执行用户','Please enter execute user','执行用户','executeUser','Input',1,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +insert into `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (44,'Filter','请填写过滤条件','Please enter filter','过滤条件','Filter','Input',0,NULL,NULL,0,NULL,0,1,1,0,'runtime'); +INSERT INTO `dss_workflow_node_ui`(`id`,`key`,`description`,`description_en`,`lable_name`,`lable_name_en`,`ui_type`,`required`,`value`,`default_value`,`is_hidden`,`condition`,`is_advanced`,`order`,`node_menu_type`,`is_base_info`,`position`) values (45,'ReuseEngine','请选择是否复用引擎','Please choose to reuse engin or not','是否复用引擎','reuse-engine-or-not','Select',1,'[\"true\",\"false\"]','true',0,NULL,0,1,1,0,'startup'); + + +DELETE FROM dss_workflow_node_ui_to_validate; +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (31,31); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (32,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (32,40); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (32,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (28,28); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (27,27); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (3,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (3,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (4,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (4,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (22,44); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (22,45); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (22,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (25,25); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (26,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (26,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (21,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (21,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (21,46); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (29,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (29,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (23,47); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (23,48); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (24,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (24,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (7,7); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (8,8); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (9,9); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (10,10); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (11,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (11,57); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (1,48); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (1,47); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (2,48); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (2,47); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (13,13); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (14,47); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (16,50); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (17,50); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (18,50); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (19,51); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (19,47); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (5,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (6,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (12,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (20,53); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (36,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (30,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (33,54); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (34,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (35,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (31,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (32,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (34,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (22,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (20,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (26,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (21,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (25,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (1,56); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (2,56); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (13,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (16,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (19,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (36,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (14,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (37,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (1,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (2,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (39,32); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (39,41); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (39,46); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (39,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (40,47); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (40,48); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (23,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (40,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (15,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (22,58); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (26,58); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (41,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (43,55); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (44,52); +insert into `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) values (42,52); +INSERT INTO `dss_workflow_node_ui_to_validate`(`ui_id`,`validate_id`) VALUES (45,59); + +DELETE FROM dss_workflow_node_ui_validate; +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('7','NumInterval','[1,15]','驱动器内存大小，默认值：2','Drive memory size, default value: 2','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('8','NumInterval','[3,15]','执行器内存大小，默认值：3','Actuator memory size, default value: 3','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('9','NumInterval','[1,10]','执行器核心个数，默认值：1','Number of cores per executor, default value : 1','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('10','NumInterval','[1,40]','执行器个数，默认值：2','Number of per executor, default value : 2','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('13','OFT','[\"node\"]','请选择类型','Please select type','change'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('25','OFT','[\"RECEIVE\"]','','Please select ','change'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('27','NumInterval','[1,1000]','请填写查询频率，默认10次，范围：1-1000','Please fill in the inquiry frequency, default : 10, range is 1 to 1000','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('28','NumInterval','[1,1000]','请填写等待时间，默认1小时，范围：1-1000','Please enter waiting time, 1 hour by default, range is 1 to 1000','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('31','OFT','[\"hivedb\",\"maskdb\"]','','Invalid format,example:ProjectName@WFName@jobName','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('32','Regex','^[^\\u4e00-\\u9fa5]+$','此值不能输入中文','Chinese characters are not allowed','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('40','Regex','^[a-zA-Z]([^.]*\\.[^.]*){1,}$','需要检查的数据源dbname.tablename{partition}','Checked data source dbname.tablename{partition}','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('41','Regex','^.{1,500}$','长度在1到500个字符','The length is between 1 and 500 characters','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('44','Regex','^[a-zA-Z][a-zA-Z0-9_@-]*$','必须以字母开头，且只支持字母、数字、下划线、@、中横线','Started with alphabetic characters, only alphanumeric characters, underscore(_), @ and hyphen(-) are allowed','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('45','Regex','^[a-zA-Z0_9-]([^@]*@[^@]*){2}[a-zA-Z\\d]$','此值格式错误，例如：ProjectName@WFName@jobName','Invalid format,example:ProjectName@WFName@jobName','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('46','Regex','^[a-zA-Z0_9-]([^_]*_[^_]*){2}[a-zA-Z\\d]$','此值格式错误，例如：bdp_tac_name','Invalid format,example:bdp_tac_name','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('47','Regex','^.{1,128}$','长度在1到128个字符','The length is between 1 and 128 characters','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('48','Regex','^[a-zA-Z][a-zA-Z0-9_-]*$','必须以字母开头，且只支持字母、数字、下划线！','Started with alphabetic characters, only alphanumeric and underscore are allowed!','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('49','Regex','^[a-zA-Z0-9_\\u4e00-\\u9fa5]*$','只支持中文、字母、数字和下划线！','Only Chinese characters, alphanumeric characters and underscore are allowed in subject!','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('50','Regex','^[a-z][a-zA-Z0-9_.@;]*$','必须以字母开头，且只支持字母、数字、下划线、@、点','Must start with a letter and only letters, numbers, underscores, @, points are supported','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('51','Regex','^[0-9_.]*$','只支持数字、下划线、点','Only numbers, underscores and dots are supported','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('52','None',NULL,NULL,NULL,'blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('53','OFT','[\"SEND\"]',NULL,NULL,'change'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('54','NumInterval','[1,1000]','请填写等待时间，默认1小时，范围：1-1000','Please fill in the waiting time, default 1 hour, range: 1-1000','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('55','Required',NULL,'该值不能为空','The value cannot be empty\n\n','change'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('56','Function','validatorTitle','节点名不能和工作流名一样','The node name cannot be the same as the workflow name',NULL); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('57','Regex','^[a-zA-Z][a-zA-Z0-9_.-]*$','必须以字母开头，且只支持字母、数字、下划线、点！','It must start with a letter and only supports letters, numbers, underscores and dots!','blur'); +insert into `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('58','Regex','(.+)@(.+)@(.+)','此格式错误，例如：ProjectName@WFName@jobName','Invalid format,example:ProjectName@WFName@jobName','blur'); +INSERT INTO `dss_workflow_node_ui_validate` (`id`, `validate_type`, `validate_range`, `error_msg`, `error_msg_en`, `trigger`) values('59','OFT','["true","false"]','请填写是否复用引擎，false：不复用，true：复用','Please fill in whether or not to reuse engine, true: reuse, false: not reuse','blur'); + +DELETE FROM dss_appconn; +INSERT INTO `dss_appconn` (`id`, `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) VALUES (2, 'orchestrator-framework', 0, 1, NULL, 0, NULL, 'com.webank.wedatasphere.dss.appconn.orchestrator.DefaultOrchestratorFrameworkAppConn', 'DSS_INSTALL_HOME_VAL/dss-appconns/orchestrator-framework/lib', ''); +INSERT INTO `dss_appconn` (`id`, `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) VALUES (3, 'workflow', 0, 1, NULL, 0, NULL, 'com.webank.wedatasphere.dss.appconn.workflow.DefaultWorkflowAppConn', 'DSS_INSTALL_HOME_VAL/dss-appconns/workflow/lib', ''); +INSERT INTO `dss_appconn` (`id`, `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) VALUES (5, 'eventchecker', 0, 1, NULL, 0, NULL, 'com.webank.wedatasphere.dss.appconn.eventchecker.EventCheckerAppConn', 'DSS_INSTALL_HOME_VAL/dss-appconns/eventchecker/lib', NULL); +INSERT INTO `dss_appconn` (`id`, `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) VALUES (6, 'datachecker', 0, 1, NULL, 0, NULL, 'com.webank.wedatapshere.dss.appconn.datachecker.DataCheckerAppConn', 'DSS_INSTALL_HOME_VAL/dss-appconns/datachecker/lib', NULL); +insert into `dss_appconn` (`id`, `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) values('7','sendemail','0','1',NULL,'0',NULL,'com.webank.wedatasphere.dss.appconn.sendemail.SendEmailAppConn','DSS_INSTALL_HOME_VAL/dss-appconns/sendemail/lib',NULL); + + +select @dss_appconn_orchestratorId:=id from `dss_appconn` where `appconn_name` = 'orchestrator-framework'; +select @dss_appconn_workflowId:=id from `dss_appconn` where `appconn_name` = 'workflow'; + +select @dss_appconn_eventcheckerId:=id from `dss_appconn` where `appconn_name` = 'eventchecker'; +select @dss_appconn_datacheckerId:=id from `dss_appconn` where `appconn_name` = 'datachecker'; + +DELETE FROM dss_appconn_instance; +INSERT INTO `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) VALUES (@dss_appconn_orchestratorId, 'DEV', 'http://ORCHESTRATOR_IP:ORCHESTRATOR_PORT/#/workspaceHome?workspaceId=104', '', 'http://ORCHESTRATOR_IP:ORCHESTRATOR_PORT/#/workspaceHome?workspaceId=104', 'http://ORCHESTRATOR_IP:ORCHESTRATOR_PORT/#/workspaceHome?workspaceId=104'); +INSERT INTO `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) VALUES (@dss_appconn_workflowId, 'DEV', 'http://WORKFLOW_IP:WORKFLOW_PORT/#/workspaceHome?workspaceId=104', '', 'http://WORKFLOW_IP:WORKFLOW_PORT/#/workspaceHome?workspaceId=104', 'http://WORKFLOW_IP:WORKFLOW_PORT/#/workspaceHome?workspaceId=104'); +INSERT INTO `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) VALUES (@dss_appconn_eventcheckerId, 'DEV', 'eventchecker', '{"msg.eventchecker.jdo.option.name": "msg","msg.eventchecker.jdo.option.url": "EVENTCHECKER_JDBC_URL","msg.eventchecker.jdo.option.username": "EVENTCHECKER_JDBC_USERNAME","msg.eventchecker.jdo.option.password": "EVENTCHECKER_JDBC_PASSWORD"}', NULL, NULL); +INSERT INTO `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) VALUES (@dss_appconn_datacheckerId, 'DEV', 'datachecker', '{"job.datachecker.jdo.option.name":"job","job.datachecker.jdo.option.url":"DATACHECKER_JOB_JDBC_URL","job.datachecker.jdo.option.username":"DATACHECKER_JOB_JDBC_USERNAME","job.datachecker.jdo.option.password":"DATACHECKER_JOB_JDBC_PASSWORD","bdp.datachecker.jdo.option.name":"bdp","bdp.datachecker.jdo.option.url":"DATACHECKER_BDP_JDBC_URL","bdp.datachecker.jdo.option.username":"DATACHECKER_BDP_JDBC_USERNAME","bdp.datachecker.jdo.option.password":"DATACHECKER_BDP_JDBC_PASSWORD","bdp.datachecker.jdo.option.login.type":"base64","bdp.mask.url":"http://BDP_MASK_IP:BDP_MASK_PORT/api/v1/mask-status?","bdp.mask.app.id":"wtss","bdp.mask.app.token":"20a0ccdfc0"}', NULL, NULL); +insert into `dss_appconn_instance` (`id`, `appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) values('12','7','DEV','sendemail','{"email.host":"EMAIL_HOST","email.port":"EMAIL_PORT","email.username":"EMAIL_USERNAME","email.password":"EMAIL_PASSWORD","email.protocol":"EMAIL_PROTOCOL"}',NULL,NULL); + +DELETE FROM dss_component_role; +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',5,'6','0',now(),'system'); + +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',7,'6','0',now(),'system'); + +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',10,'6','0',now(),'system'); + +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1','11','6','0',now(),'system'); + +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',14,'6','0',now(),'system'); + +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',16,'6','0',now(),'system'); + +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'7','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'1','1',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'2','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'3','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'4','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'5','0',now(),'system'); +INSERT INTO `dss_component_role` (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) VALUES('-1',17,'6','0',now(),'system'); diff --git a/db/qualitis.sql b/db/qualitis.sql deleted file mode 100644 index 04fb15cfb..000000000 --- a/db/qualitis.sql +++ /dev/null @@ -1,3 +0,0 @@ -INSERT INTO `dss_application` (`id`, `name`, `url`, `is_user_need_init`, `level`, `user_init_url`, `exists_project_service`, `project_url`, `enhance_json`, `if_iframe`, `homepage_url`, `redirect_url`) VALUES (NULL, 'qualitis', 'http://QUALITIS_ADRESS_IP_2:QUALITIS_ADRESS_PORT', '0', '1', NULL, '1', 'http://QUALITIS_ADRESS_IP_2:QUALITIS_ADRESS_PORT/#/projects/list?id=${projectId}&flow=true', NULL, '1', 'http://QUALITIS_ADRESS_IP_2:QUALITIS_ADRESS_PORT/#/dashboard', 'http://QUALITIS_ADRESS_IP_2:QUALITIS_ADRESS_PORT/qualitis/api/v1/redirect'); -SELECT @qualitis_appid:=id from dss_application WHERE `name` = 'qualitis'; -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.qualitis', @qualitis_appid, NULL, '1', '0', '1', 'http://QUALITIS_ADRESS_IP_2:QUALITIS_ADRESS_PORT/#/addGroupTechniqueRule?tableType=1&id=${projectId}&ruleGroupId=${ruleGroupId}&nodeId=${nodeId}'); \ No newline at end of file diff --git a/db/visualis.sql b/db/visualis.sql deleted file mode 100644 index 71070c7bd..000000000 --- a/db/visualis.sql +++ /dev/null @@ -1,34 +0,0 @@ -INSERT INTO `dss_application` (`id`, `name`, `url`, `is_user_need_init`, `level`, `user_init_url`, `exists_project_service`, `project_url`, `enhance_json`, `if_iframe`, `homepage_url`, `redirect_url`) VALUES (NULL, 'visualis', null, '0', '1', NULL, '0', NULL, NULL, '1', NULL, NULL); -SELECT @visualis_appid:=id from dss_application WHERE `name` = 'visualis'; -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.visualis.display', @visualis_appid, '1', '1', '1', '1', NULL); -INSERT INTO `dss_workflow_node` (`id`, `icon`, `node_type`, `application_id`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `support_jump`, `jump_url`) VALUES (NULL, NULL, 'linkis.appjoint.visualis.dashboard', @visualis_appid, '1', '1', '1', '1', NULL);UPDATE `dss_application` SET url = 'http://VISUALIS_NGINX_IP_2:VISUALIS_NGINX_PORT' WHERE `name` in('visualis'); -UPDATE `dss_application` SET url = 'http://VISUALIS_NGINX_IP_2:VISUALIS_NGINX_PORT' WHERE `name` in('visualis'); -UPDATE `dss_application` SET project_url = 'http://VISUALIS_NGINX_IP_2:VISUALIS_NGINX_PORT/dss/visualis/#/project/${projectId}',homepage_url = 'http://VISUALIS_NGINX_IP_2:VISUALIS_NGINX_PORT/dss/visualis/#/projects' WHERE `name` in('visualis'); -UPDATE `dss_workflow_node` SET jump_url = 'http://VISUALIS_NGINX_IP_2:VISUALIS_NGINX_PORT/dss/visualis/#/project/${projectId}/display/${nodeId}' where node_type = 'linkis.appjoint.visualis.display'; -UPDATE `dss_workflow_node` SET jump_url = 'http://VISUALIS_NGINX_IP_2:VISUALIS_NGINX_PORT/dss/visualis/#/project/${projectId}/portal/${nodeId}/portalName/${nodeName}' where node_type = 'linkis.appjoint.visualis.dashboard'; -INSERT INTO `linkis_application` (`id`, `name`, `chinese_name`, `description`) VALUES (NULL, 'visualis', NULL, NULL); -select @application_id:=id from `linkis_application` where `name` = 'visualis'; - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dss-appconn/appconns/dss-datachecker-appconn/pom.xml b/dss-appconn/appconns/dss-datachecker-appconn/pom.xml new file mode 100644 index 000000000..c970ff2b4 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/pom.xml @@ -0,0 +1,158 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-datachecker-appconn + + + + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-development-process-standard-execution + ${dss.version} + + + + log4j + log4j + 1.2.17 + + + + com.squareup.okhttp3 + okhttp + 4.2.2 + + + + org.apache.commons + commons-lang3 + 3.4 + + + + com.alibaba + druid + 1.0.28 + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + + + + src/main/resources + + **/*.properties + **/application.yml + **/bootstrap.yml + **/log4j2.xml + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-datachecker-appconn/src/main/assembly/distribution.xml new file mode 100644 index 000000000..3863a4afe --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/assembly/distribution.xml @@ -0,0 +1,66 @@ + + + + dss-datachecker-appconn + + dir + + true + datachecker + + + + + + lib + true + true + false + true + true + + + + + + ${basedir}/src/main/resources + + appconn.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + log4j.properties + log4j2.xml + + 0777 + conf + unix + + + + + + diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataChecker.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataChecker.java new file mode 100644 index 000000000..dfd2ee63d --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataChecker.java @@ -0,0 +1,93 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker; + + +import com.webank.wedatapshere.dss.appconn.datachecker.connector.DataCheckerDao; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionState; +import org.apache.log4j.Logger; + +import java.util.Properties; + +public class DataChecker { + public final static String SOURCE_TYPE = "source.type"; + public final static String DATA_OBJECT = "check.object"; + public final static String WAIT_TIME = "max.check.hours"; + public final static String QUERY_FREQUENCY = "query.frequency"; + public final static String TIME_SCAPE = "time.scape"; + public final static String MASK_URL = "bdp.mask.url"; + public final static String MASK_APP_ID = "bdp.mask.app.id"; + public final static String MASK_APP_TOKEN = "bdp.mask.app.token"; + + private Properties p; + private static final Logger logger = Logger.getRootLogger(); + DataCheckerDao wbDao = DataCheckerDao.getInstance(); + DataCheckerExecutionAction dataCheckerAction = null; + public long maxWaitTime; + public int queryFrequency; + + public DataChecker(Properties p, DataCheckerExecutionAction action) { + this.p = p; + dataCheckerAction = action; + maxWaitTime = Long.valueOf(p.getProperty(DataChecker.WAIT_TIME, "1")) * 3600 * 1000; + //test over time +// maxWaitTime = Long.valueOf(p.getProperty(DataChecker.WAIT_TIME, "1")) * 120 * 1000; + queryFrequency = Integer.valueOf(p.getProperty(DataChecker.QUERY_FREQUENCY, "30000")); + + } + + public void run() { + dataCheckerAction.setState(RefExecutionState.Running); + try { + if(p == null) { + throw new RuntimeException("Properties is null. Can't continue"); + } + if (!p.containsKey(SOURCE_TYPE)) { + logger.info("Properties " + SOURCE_TYPE + " value is Null !"); + } + if (!p.containsKey(DATA_OBJECT)) { + logger.info("Properties " + DATA_OBJECT + " value is Null !"); + } + begineCheck(dataCheckerAction); + }catch (Exception ex){ + dataCheckerAction.setState(RefExecutionState.Failed); + throw new RuntimeException("get DataChecker result failed", ex); + } + + } + + public void begineCheck(RefExecutionAction action){ + boolean success=false; + try { + success= wbDao.validateTableStatusFunction(p, logger,action); + }catch (Exception ex){ + dataCheckerAction.setState(RefExecutionState.Failed); + logger.error("datacheck error",ex); + throw new RuntimeException("get DataChecker result failed", ex); + } + if(success) { + dataCheckerAction.setState(RefExecutionState.Success); + }else { + dataCheckerAction.setState(RefExecutionState.Running); + } + } + + public void cancel() { + } + +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerAppConn.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerAppConn.java new file mode 100644 index 000000000..61d48e28c --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerAppConn.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker; + +import com.webank.wedatapshere.dss.appconn.datachecker.standard.DataCheckerDevelopmentStandard; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractAppConn; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; + +public class DataCheckerAppConn extends AbstractAppConn implements OnlyDevelopmentAppConn { + + private DataCheckerDevelopmentStandard standard; + + @Override + protected void initialize() { + standard = new DataCheckerDevelopmentStandard(); + } + + @Override + public DevelopmentIntegrationStandard getOrCreateDevelopmentStandard() { + return standard; + } +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerCompletedExecutionResponseRef.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerCompletedExecutionResponseRef.java new file mode 100644 index 000000000..88c6bc619 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerCompletedExecutionResponseRef.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker; + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; + +import java.util.Map; + +public class DataCheckerCompletedExecutionResponseRef extends CompletedExecutionResponseRef { + + private Exception exception; + public void setException(Exception exception) { + this.exception = exception; + } + + + + public DataCheckerCompletedExecutionResponseRef(int status) { + super(status); + } + + public DataCheckerCompletedExecutionResponseRef(String responseBody, int status) { + super(responseBody, status); + } + + public void setStatus(int status){ + this.status = status; + } + + + @Override + public Map toMap() { + return null; + } + +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/common/MaskCheckNotExistException.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/common/MaskCheckNotExistException.java new file mode 100644 index 000000000..5f1443674 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/common/MaskCheckNotExistException.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker.common; + +public class MaskCheckNotExistException extends Exception { + + public MaskCheckNotExistException(final String message) { + super(message); + } +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/connector/DataCheckerDao.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/connector/DataCheckerDao.java new file mode 100644 index 000000000..6ff9387d7 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/connector/DataCheckerDao.java @@ -0,0 +1,367 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker.connector; + +import com.alibaba.druid.pool.DruidDataSource; + +import com.webank.wedatapshere.dss.appconn.datachecker.DataChecker; +import com.webank.wedatapshere.dss.appconn.datachecker.common.MaskCheckNotExistException; +import com.webank.wedatapshere.dss.appconn.datachecker.utils.HttpUtils; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; +import okhttp3.FormBody; +import okhttp3.RequestBody; +import okhttp3.Response; +import okhttp3.ResponseBody; +import org.apache.commons.lang.StringUtils; +import org.apache.log4j.Logger; + +import javax.sql.DataSource; +import java.io.IOException; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.function.Predicate; +import java.util.function.Supplier; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +public class DataCheckerDao { + + private static final String SQL_SOURCE_TYPE_JOB_TABLE = + "SELECT * FROM DBS d JOIN TBLS t ON t.DB_ID = d.DB_ID WHERE d.NAME=? AND t.TBL_NAME=?"; + + private static final String SQL_SOURCE_TYPE_JOB_PARTITION = + "SELECT * FROM DBS d JOIN TBLS t ON t.DB_ID = d.DB_ID JOIN PARTITIONS p ON p.TBL_ID = t.TBL_ID WHERE d.NAME=? AND t.TBL_NAME=? AND p.PART_NAME=?"; + + private static final String SQL_SOURCE_TYPE_BDP = + "SELECT * FROM desktop_bdapimport WHERE bdap_db_name = ? AND bdap_table_name = ? AND target_partition_name = ? AND status = '1';"; + + private static final String SQL_SOURCE_TYPE_BDP_WITH_TIME_CONDITION = + "SELECT * FROM desktop_bdapimport WHERE bdap_db_name = ? AND bdap_table_name = ? AND target_partition_name = ? " + + "AND (UNIX_TIMESTAMP() - UNIX_TIMESTAMP(STR_TO_DATE(modify_time, '%Y-%m-%d %H:%i:%s'))) <= ? AND status = '1';"; + + private static DataSource jobDS; + private static DataSource bdpDS; + private static DataCheckerDao instance; + + public static DataCheckerDao getInstance() { + if (instance == null) { + synchronized (DataCheckerDao.class) { + if (instance == null) { + instance = new DataCheckerDao(); + } + } + } + return instance; + } + + public boolean validateTableStatusFunction(Properties props, Logger log, RefExecutionAction action) { + if (jobDS == null) { + jobDS = DataDruidFactory.getJobInstance(props, log); + if (jobDS == null) { + log.error("Error getting Druid DataSource instance"); + return false; + } + } + if (bdpDS == null) { + bdpDS = DataDruidFactory.getBDPInstance(props, log); + if (bdpDS == null) { + log.warn("Error getting Druid DataSource instance"); + return false; + } + } + removeBlankSpace(props); + log.info("=============================Data Check Start=========================================="); + String dataCheckerInfo = props.getProperty(DataChecker.DATA_OBJECT); + if(null!=action.getExecutionRequestRefContext()) { + action.getExecutionRequestRefContext().appendLog("Database table partition info : " + dataCheckerInfo); + } + log.info("(DataChecker info) database table partition info : " + dataCheckerInfo); + long waitTime = Long.valueOf(props.getProperty(DataChecker.WAIT_TIME, "1")) * 3600 * 1000; + int queryFrequency = Integer.valueOf(props.getProperty(DataChecker.QUERY_FREQUENCY, "30000")); + log.info("(DataChecker info) wait time : " + waitTime); + log.info("(DataChecker info) query frequency : " + queryFrequency); + List> dataObjectList = extractProperties(props); + try (Connection jobConn = jobDS.getConnection(); + Connection bdpConn = bdpDS.getConnection()) { + boolean flag = dataObjectList + .stream() + .allMatch(proObjectMap -> getDataCheckResult(proObjectMap, jobConn, bdpConn, props, log)); + if (flag) { + log.info("=============================Data Check End=========================================="); + return true; + } + + } catch (SQLException e) { + throw new RuntimeException("get DataChecker result failed", e); + } + + log.info("=============================Data Check End=========================================="); + return false; + } + + private boolean getDataCheckResult(Map proObjectMap, Connection jobConn, Connection bdpConn, Properties props, Logger log) { + Predicate> hasDataSource = p -> { + if (StringUtils.isEmpty(proObjectMap.get(DataChecker.SOURCE_TYPE))) { + return false; + } else { + return true; + } + }; + Predicate> hasNotDataSource = hasDataSource.negate(); + Supplier sourceType = () -> proObjectMap.get(DataChecker.SOURCE_TYPE).toLowerCase(); + Predicate> isJobDataSource = p -> sourceType.get().equals("hivedb") || sourceType.get().equals("job"); + Predicate> isBdpDataSource = p -> sourceType.get().equals("maskdb") || sourceType.get().equals("bdp"); + Predicate> isOdsDB = p -> { + String dataObject = proObjectMap.get(DataChecker.DATA_OBJECT) + .replace(" ", "").trim(); + String dbName = dataObject.split("\\.")[0]; + return dbName.contains("_ods"); + }; + Predicate> isNotOdsDB = isOdsDB.negate(); + Predicate> isCheckMetadata = (hasDataSource.and(isJobDataSource)).or(hasNotDataSource.and(isNotOdsDB)); + Predicate> isCheckMask = (hasDataSource.and(isBdpDataSource)).or(hasNotDataSource.and(isOdsDB)); + return isCheckMetadata.test(proObjectMap) + ? getJobTotalCount(proObjectMap, jobConn, log) > 0 + : isCheckMask.test(proObjectMap) && + (getBdpTotalCount(proObjectMap, bdpConn, log, props) > 0 + || "success".equals(fetchMaskCode(proObjectMap, log, props).get("maskStatus"))); + } + + private void sleep(long sleepTime) { + try { + Thread.sleep(sleepTime); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + + private void removeBlankSpace(Properties props) { + try { + props.entrySet().forEach(entry -> { + String value = entry.getValue().toString().replaceAll(" ", "").trim(); + entry.setValue(value); + }); + } catch (Exception e) { + throw new RuntimeException("remove job space char failed", e); + } + } + + private List> extractProperties(Properties p) { + return p.keySet().stream() + .map(key -> key2Map(key, p)).filter(x -> x.size() > 0) + .collect(Collectors.toList()); + } + + private Map key2Map(Object key, Properties p) { + Map proMap = new HashMap<>(); + String skey = String.valueOf(key); + if (skey.contains(DataChecker.DATA_OBJECT)) { + String[] keyArr = skey.split("\\."); + if (keyArr.length == 3) { + String keyNum = keyArr[2]; + String stKey = DataChecker.SOURCE_TYPE + "." + keyNum; + String doKey = DataChecker.DATA_OBJECT + "." + keyNum; + if (null != p.get(stKey)) { + proMap.put(DataChecker.SOURCE_TYPE, String.valueOf(p.get(stKey))); + } + proMap.put(DataChecker.DATA_OBJECT, String.valueOf(p.get(doKey))); + } else { + String stKey = DataChecker.SOURCE_TYPE; + String doKey = DataChecker.DATA_OBJECT; + if (null != p.get(stKey)) { + proMap.put(DataChecker.SOURCE_TYPE, String.valueOf(p.get(stKey))); + } + proMap.put(DataChecker.DATA_OBJECT, String.valueOf(p.get(doKey))); + } + } + + return proMap; + } + + private PreparedStatement getJobStatement(Connection conn, String dataObject) throws SQLException { + String dataScape = dataObject.contains("{") ? "Partition" : "Table"; + String[] dataObjectArray = dataObject.split("\\."); + String dbName = dataObject.split("\\.")[0]; + String tableName = dataObject.split("\\.")[1]; + if (dataScape.equals("Partition")) { + Pattern pattern = Pattern.compile("\\{([^\\}]+)\\}"); + Matcher matcher = pattern.matcher(dataObject); + String partitionName = null; + if (matcher.find()) { + partitionName = matcher.group(1); + } + partitionName = partitionName.replace("\'", "").replace("\"", ""); + tableName = tableName.split("\\{")[0]; + PreparedStatement pstmt = conn.prepareCall(SQL_SOURCE_TYPE_JOB_PARTITION); + pstmt.setString(1, dbName); + pstmt.setString(2, tableName); + pstmt.setString(3, partitionName); + return pstmt; + } else if (dataObjectArray.length == 2) { + PreparedStatement pstmt = conn.prepareCall(SQL_SOURCE_TYPE_JOB_TABLE); + pstmt.setString(1, dbName); + pstmt.setString(2, tableName); + return pstmt; + } else { + throw new SQLException("Error for DataObject format!"); + } + } + + private PreparedStatement getBdpStatement(Connection conn, String dataObject, String timeScape) throws SQLException { + String dataScape = dataObject.contains("{") ? "Partition" : "Table"; + String dbName = dataObject.split("\\.")[0]; + String tableName = dataObject.split("\\.")[1]; + String partitionName = ""; + Pattern pattern = Pattern.compile("\\{([^\\}]+)\\}"); + if (dataScape.equals("Partition")) { + Matcher matcher = pattern.matcher(dataObject); + if (matcher.find()) { + partitionName = matcher.group(1); + } + partitionName = partitionName.replace("\'", "").replace("\"", ""); + tableName = tableName.split("\\{")[0]; + } + PreparedStatement pstmt = null; + if (timeScape.equals("NULL")) { + pstmt = conn.prepareCall(SQL_SOURCE_TYPE_BDP); + } else { + pstmt = conn.prepareCall(SQL_SOURCE_TYPE_BDP_WITH_TIME_CONDITION); + pstmt.setInt(4, Integer.valueOf(timeScape) * 3600); + } + pstmt.setString(1, dbName); + pstmt.setString(2, tableName); + pstmt.setString(3, partitionName); + return pstmt; + } + + private long getJobTotalCount(Map proObjectMap, Connection conn, Logger log) { + String dataObject = proObjectMap.get(DataChecker.DATA_OBJECT); + if (dataObject != null) { + dataObject = dataObject.replace(" ", "").trim(); + } + log.info("-------------------------------------- search hive/spark/mr data "); + log.info("-------------------------------------- : " + dataObject); + try (PreparedStatement pstmt = getJobStatement(conn, dataObject)) { + ResultSet rs = pstmt.executeQuery(); + return rs.last() ? rs.getRow() : 0; + } catch (SQLException e) { + log.error("fetch data from Hive MetaStore error", e); + return 0; + } + } + + private long getBdpTotalCount(Map proObjectMap, Connection conn, Logger log, Properties props) { + String dataObject = proObjectMap.get(DataChecker.DATA_OBJECT); + if (dataObject != null) { + dataObject = dataObject.replace(" ", "").trim(); + } + String timeScape = props.getOrDefault(DataChecker.TIME_SCAPE, "NULL").toString(); + log.info("-------------------------------------- search bdp data "); + log.info("-------------------------------------- : " + dataObject); + try (PreparedStatement pstmt = getBdpStatement(conn, dataObject, timeScape)) { + ResultSet rs = pstmt.executeQuery(); + return rs.last() ? rs.getRow() : 0; + } catch (SQLException e) { + log.error("fetch data from Hive MetaStore error", e); + return 0; + } + } + + private Map fetchMaskCode(Map proObjectMap, Logger log, Properties props) { + log.info("=============================调用BDP MASK接口查询数据状态=========================================="); + Map resultMap = new HashMap(); + String maskUrl = props.getProperty(DataChecker.MASK_URL); + String dataObject = proObjectMap.get(DataChecker.DATA_OBJECT); + if (dataObject != null) { + dataObject = dataObject.replace(" ", "").trim(); + } + String dataScape = dataObject.contains("{") ? "Partition" : "Table"; + String dbName = dataObject.split("\\.")[0]; + String tableName = dataObject.split("\\.")[1]; + String partitionName = ""; + Pattern pattern = Pattern.compile("\\{([^\\}]+)\\}"); + if (dataScape.equals("Partition")) { + Matcher matcher = pattern.matcher(dataObject); + if (matcher.find()) { + partitionName = matcher.group(1); + } + partitionName = partitionName.replace("\'", "").replace("\"", ""); + tableName = tableName.split("\\{")[0]; + } + try { + RequestBody requestBody = new FormBody.Builder() + .add("targetDb", dbName) + .add("targetTable", tableName) + .add("partition", partitionName) + .build(); + Map dataMap = HttpUtils.initSelectParams(props); + log.info("request body:dbName--" + dbName + " tableName--" + tableName + " partitionName--" + partitionName); + Response response = HttpUtils.httpClientHandleBase(maskUrl, requestBody, dataMap); + handleResponse(response, resultMap, log); + } catch (IOException e) { + log.error("fetch data from BDP MASK failed "); + resultMap.put("maskStatus", "noPrepare"); + } catch (MaskCheckNotExistException e) { + String errorMessage = "fetch data from BDP MASK failed" + + "please check database: " + dbName + ",table: " + tableName + "is exist"; + log.error(errorMessage); + throw new RuntimeException(errorMessage, e); + } + return resultMap; + } + + private void handleResponse(Response response, Map proObjectMap, Logger log) + throws IOException, MaskCheckNotExistException { + int responseCode = response.code(); + ResponseBody body = response.body(); + if (responseCode == 200) { + handleResponseBody(body, proObjectMap, log); + } else { + proObjectMap.put("maskStatus", "noPrepare"); + } + } + + private void handleResponseBody(ResponseBody body, Map proObjectMap, Logger log) + throws IOException, MaskCheckNotExistException { + String bodyStr = body.string(); + log.info("mask interface response body：" + bodyStr); + Map entityMap = HttpUtils.getReturnMap(bodyStr); + String codeValue = (String) entityMap.get("code"); + if ("200".equals(codeValue)) { + proObjectMap.put("maskStatus", "success"); + } else if ("1011".equals(codeValue)) { + throw new MaskCheckNotExistException("Mask check failed"); + } else { + proObjectMap.put("maskStatus", "noPrepare"); + } + } + + public static void closeDruidDataSource() { + DruidDataSource jobDSObject = (DruidDataSource) jobDS; + if (jobDSObject != null) { + jobDSObject.close(); + } + } + +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/connector/DataDruidFactory.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/connector/DataDruidFactory.java new file mode 100644 index 000000000..ba7e9adba --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/connector/DataDruidFactory.java @@ -0,0 +1,157 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker.connector; + +import com.alibaba.druid.pool.DruidDataSource; +import org.apache.commons.lang3.StringUtils; +import org.apache.log4j.Logger; + +import java.util.Base64; +import java.util.Properties; + +public class DataDruidFactory { + private static DruidDataSource jobInstance; + private static DruidDataSource bdpInstance; + private static DruidDataSource msgInstance; + + public static DruidDataSource getJobInstance(Properties props, Logger log) { + if (jobInstance == null ) { + synchronized (DataDruidFactory.class) { + if(jobInstance == null) { + try { + jobInstance = createDataSource(props, log, "Job"); + } catch (Exception e) { + throw new RuntimeException("Error creating Druid DataSource", e); + } + } + } + } + return jobInstance; + } + public static DruidDataSource getBDPInstance(Properties props, Logger log) { + if (bdpInstance == null ) { + synchronized (DataDruidFactory.class) { + if(bdpInstance == null) { + try { + bdpInstance = createDataSource(props, log, "BDP"); + } catch (Exception e) { + throw new RuntimeException("Error creating Druid DataSource", e); + } + } + } + } + return bdpInstance; + } + + public static DruidDataSource getMsgInstance(Properties props, Logger log) { + if (msgInstance == null ) { + synchronized (DataDruidFactory.class) { + if(msgInstance == null) { + try { + msgInstance = createDataSource(props, log, "Msg"); + } catch (Exception e) { + throw new RuntimeException("Error creating Druid DataSource", e); + } + } + } + } + return msgInstance; + } + + private static DruidDataSource createDataSource(Properties props, Logger log, String type) { + String name = null; + String url = null; + String username = null; + String password = null; + String loginType = null; + if (type.equals("Job")) { + name = props.getProperty("job.datachecker.jdo.option.name"); + url = props.getProperty("job.datachecker.jdo.option.url"); + username = props.getProperty("job.datachecker.jdo.option.username"); + password = props.getProperty("job.datachecker.jdo.option.password"); + loginType = props.getProperty("job.datachecker.jdo.option.login.type"); + log.info("job url is:"+url+"and name is:"+username); + try { + if("base64".equals(loginType)) { + password = new String(Base64.getDecoder().decode(props.getProperty("job.datachecker.jdo.option.password").getBytes()), "UTF-8"); + }else { + password = props.getProperty("job.datachecker.jdo.option.password"); + } + } catch (Exception e){ + log.error("password decore failed" + e); + } + }else if (type.equals("BDP")) { + name = props.getProperty("bdp.datachecker.jdo.option.name"); + url = props.getProperty("bdp.datachecker.jdo.option.url"); + username = props.getProperty("bdp.datachecker.jdo.option.username"); + password = props.getProperty("bdp.datachecker.jdo.option.password"); + loginType = props.getProperty("bdp.datachecker.jdo.option.login.type"); + log.info("bdp url is:"+url+"and name is:"+username); + try { + if("base64".equals(loginType)) { + password = new String(Base64.getDecoder().decode(props.getProperty("bdp.datachecker.jdo.option.password").getBytes()), "UTF-8"); + }else { + password = props.getProperty("bdp.datachecker.jdo.option.password"); + } + } catch (Exception e){ + log.error("password decore failed" + e); + } + } + int initialSize = Integer.valueOf(props.getProperty("datachecker.jdo.option.initial.size", "1")); + int maxActive = Integer.valueOf(props.getProperty("datachecker.jdo.option.max.active", "100")); + int minIdle = Integer.valueOf(props.getProperty("datachecker.jdo.option.min.idle", "1")); + long maxWait = Long.valueOf(props.getProperty("datachecker.jdo.option.max.wait", "60000")); + String validationQuery = props.getProperty("datachecker.jdo.option.validation.quert", "SELECT 'x'"); + long timeBetweenEvictionRunsMillis = Long.valueOf(props.getProperty("datachecker.jdo.option.time.between.eviction.runs.millis", "6000")); + long minEvictableIdleTimeMillis = Long.valueOf(props.getProperty("datachecker.jdo.option.evictable.idle,time.millis", "300000")); + boolean testOnBorrow = Boolean.valueOf(props.getProperty("datachecker.jdo.option.test.on.borrow", "true")); + int maxOpenPreparedStatements = Integer.valueOf(props.getProperty("datachecker.jdo.option.max.open.prepared.statements", "-1")); + + + if (timeBetweenEvictionRunsMillis > minEvictableIdleTimeMillis) { + timeBetweenEvictionRunsMillis = minEvictableIdleTimeMillis; + } + + DruidDataSource ds = new DruidDataSource(); + + if (StringUtils.isNotBlank(name)) { + ds.setName(name); + } + + ds.setUrl(url); + ds.setDriverClassName("com.mysql.jdbc.Driver"); + ds.setUsername(username); + ds.setPassword(password); + ds.setInitialSize(initialSize); + ds.setMinIdle(minIdle); + ds.setMaxActive(maxActive); + ds.setMaxWait(maxWait); + ds.setTestOnBorrow(testOnBorrow); + ds.setValidationQuery(validationQuery); + ds.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis); + ds.setMinEvictableIdleTimeMillis(minEvictableIdleTimeMillis); + if (maxOpenPreparedStatements > 0) { + ds.setPoolPreparedStatements(true); + ds.setMaxPoolPreparedStatementPerConnectionSize( + maxOpenPreparedStatements); + } else { + ds.setPoolPreparedStatements(false); + } + log.info("Druid data source initialed!"); + return ds; + } +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/service/DataCheckerExecuteService.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/service/DataCheckerExecuteService.java new file mode 100644 index 000000000..3ab3e5674 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/service/DataCheckerExecuteService.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker.service; + +import com.webank.wedatapshere.dss.appconn.datachecker.DataCheckerRefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExecutionService; + +public class DataCheckerExecuteService extends AbstractRefExecutionService { + + + @Override + public RefExecutionOperation createRefExecutionOperation() { + DataCheckerRefExecutionOperation dataCheckerRefExecutionOperation = new DataCheckerRefExecutionOperation(); + dataCheckerRefExecutionOperation.setDevelopmentService(this); + return dataCheckerRefExecutionOperation; + } + +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/standard/DataCheckerDevelopmentStandard.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/standard/DataCheckerDevelopmentStandard.java new file mode 100644 index 000000000..ed7171e3d --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/standard/DataCheckerDevelopmentStandard.java @@ -0,0 +1,42 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker.standard; + +import com.webank.wedatapshere.dss.appconn.datachecker.service.DataCheckerExecuteService; +import com.webank.wedatasphere.dss.standard.app.development.standard.OnlyExecutionDevelopmentStandard; +import com.webank.wedatasphere.dss.standard.app.development.service.RefExecutionService; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class DataCheckerDevelopmentStandard extends OnlyExecutionDevelopmentStandard { + + + private static final Logger LOGGER = LoggerFactory.getLogger(DataCheckerDevelopmentStandard.class); + + + @Override + protected RefExecutionService createRefExecutionService() { + return new DataCheckerExecuteService(); + } + + @Override + public void init() { + LOGGER.info("class DataCheckerDevelopmentStandard init"); + } + + +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/utils/HttpUtils.java b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/utils/HttpUtils.java new file mode 100644 index 000000000..34208869e --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/java/com/webank/wedatapshere/dss/appconn/datachecker/utils/HttpUtils.java @@ -0,0 +1,117 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker.utils; + +import com.google.gson.Gson; +import com.google.gson.GsonBuilder; +import com.google.gson.reflect.TypeToken; + +import com.webank.wedatapshere.dss.appconn.datachecker.DataChecker; +import okhttp3.*; +import org.apache.commons.codec.digest.DigestUtils; +import org.apache.commons.lang3.RandomStringUtils; +import org.apache.log4j.Logger; + +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; +import java.util.concurrent.TimeUnit; + +public class HttpUtils { + + private static final Logger logger = Logger.getLogger(HttpUtils.class); + + public static Response httpClientHandleBase(String actionUrl, RequestBody requestBody, Map urlMap) throws IOException { + String maskUrl = actionUrl + "appid=" + urlMap.get("appid") + "&&nonce=" + urlMap.get("nonce") + + "&×tamp=" + urlMap.get("timestamp") + "&&signature=" + urlMap.get("signature"); + OkHttpClient okHttpClient = new OkHttpClient.Builder() + .connectTimeout(10, TimeUnit.SECONDS) + .writeTimeout(20, TimeUnit.SECONDS) + .readTimeout(20, TimeUnit.SECONDS) + .build(); + + logger.info("access mask URL is:"+maskUrl); + Request request = new Request.Builder() + .url(maskUrl) + .post(requestBody) + .build(); + Call call = okHttpClient.newCall(request); + Response response = call.execute(); + logger.info("mask interface response code：" + response.code()); + return response; + } + + public static String httpClientHandle(String actionUrl, RequestBody requestBody, Map urlMap) throws Exception{ + String returnData = ""; + try { + Response response = httpClientHandleBase(actionUrl, requestBody, urlMap); + returnData = response.body().string(); + logger.info("mask interface return message：" + returnData); + } catch (IOException e) { + e.printStackTrace(); + } + return returnData; + } + + public static String httpClientHandle(String actionUrl) throws Exception{ + OkHttpClient okHttpClient = new OkHttpClient.Builder() + .connectTimeout(10, TimeUnit.SECONDS) + .writeTimeout(20, TimeUnit.SECONDS) + .readTimeout(20, TimeUnit.SECONDS) + .build(); + Request request = new Request.Builder() + .url(actionUrl) + .build(); + Call call = okHttpClient.newCall(request); + String returnData = ""; + try { + Response response = call.execute(); + returnData = response.body().string(); + logger.info("interface return message：" + returnData); + } catch (IOException e) { + e.printStackTrace(); + } + return returnData; + } + + public static Map getReturnMap(String dataStr){ + Map dataMap = new HashMap<>(); + GsonBuilder gb = new GsonBuilder(); + Gson g = gb.create(); + dataMap = g.fromJson(dataStr, new TypeToken>(){}.getType()); + return dataMap; + } + + public static Map initSelectParams(Properties props){ + String appid = props.getProperty(DataChecker.MASK_APP_ID); + String token = props.getProperty(DataChecker.MASK_APP_TOKEN); + String nonce = RandomStringUtils.random(5, "0123456789"); + Long cur_time = System.currentTimeMillis() / 1000; + Map requestProperties = new HashMap<>(); + requestProperties.put("appid", appid); + requestProperties.put("nonce", nonce.toString()); + requestProperties.put("signature", getMD5(getMD5(appid + nonce.toString() + cur_time) + token)); + requestProperties.put("timestamp", cur_time.toString()); + return requestProperties; + } + + public static String getMD5(String str){ + return DigestUtils.md5Hex(str.getBytes()); + } + +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/appconn.properties b/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/appconn.properties new file mode 100644 index 000000000..ac52d9b55 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/appconn.properties @@ -0,0 +1,33 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +job.datachecker.jdo.option.name=job +job.datachecker.jdo.option.url=jdbc:mysql://127.0.0.1:3306/ +job.datachecker.jdo.option.username= +job.datachecker.jdo.option.password= +job.datachecker.jdo.option.login.type=base64 + +bdp.datachecker.jdo.option.name=bdp +bdp.datachecker.jdo.option.url= +bdp.datachecker.jdo.option.username= +bdp.datachecker.jdo.option.password= +bdp.datachecker.jdo.option.login.type=base64 + + +bdp.mask.url= +bdp.mask.app.id= +bdp.mask.app.token= + diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/log4j.properties b/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/log4j.properties new file mode 100644 index 000000000..ee8619595 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/log4j2.xml b/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/log4j2.xml new file mode 100644 index 000000000..8c40a73e8 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/resources/log4j2.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/scala/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerExecutionAction.scala b/dss-appconn/appconns/dss-datachecker-appconn/src/main/scala/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerExecutionAction.scala new file mode 100644 index 000000000..c99d9d635 --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/scala/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerExecutionAction.scala @@ -0,0 +1,42 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker + + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.{AbstractRefExecutionAction, LongTermRefExecutionAction, RefExecutionAction, RefExecutionState} + +class DataCheckerExecutionAction extends AbstractRefExecutionAction with LongTermRefExecutionAction{ + private[this] var _state: RefExecutionState = null + private var schedulerId: Int = _ + def state: RefExecutionState = _state + + def setState(value: RefExecutionState): Unit = { + _state = value + } + val response = new DataCheckerCompletedExecutionResponseRef(200) + private[this] var _dc: DataChecker = null + + def dc: DataChecker = _dc + + def setDc(value: DataChecker): Unit = { + _dc = value + } + + override def setSchedulerId(schedulerId: Int): Unit = this.schedulerId = schedulerId + + override def getSchedulerId: Int = schedulerId +} diff --git a/dss-appconn/appconns/dss-datachecker-appconn/src/main/scala/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerRefExecutionOperation.scala b/dss-appconn/appconns/dss-datachecker-appconn/src/main/scala/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerRefExecutionOperation.scala new file mode 100644 index 000000000..910623faa --- /dev/null +++ b/dss-appconn/appconns/dss-datachecker-appconn/src/main/scala/com/webank/wedatapshere/dss/appconn/datachecker/DataCheckerRefExecutionOperation.scala @@ -0,0 +1,190 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatapshere.dss.appconn.datachecker + +import java.util +import java.util.{Properties, UUID} + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.{AsyncExecutionRequestRef, AsyncExecutionResponseRef, CompletedExecutionResponseRef, RefExecutionAction, RefExecutionState} +import com.webank.wedatasphere.dss.standard.app.development.listener.core.{Killable, LongTermRefExecutionOperation, Procedure} +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService +import com.webank.wedatasphere.linkis.common.log.LogUtils +import com.webank.wedatasphere.linkis.common.utils.{Utils, VariableUtils} +import org.slf4j.LoggerFactory; + +class DataCheckerRefExecutionOperation extends LongTermRefExecutionOperation with Killable with Procedure{ + + private var service:DevelopmentService = _ + + private val logger = LoggerFactory.getLogger(classOf[DataCheckerRefExecutionOperation]) + + + + + + + protected def putErrorMsg(errorMsg: String, t: Throwable, action: DataCheckerExecutionAction): DataCheckerExecutionAction = t match { + + case t: Exception => + val response = action.response + response.setErrorMsg(errorMsg) + response.setException(t) + response.setIsSucceed(false) + action + } + + override def submit(requestRef: ExecutionRequestRef): RefExecutionAction = { + val asyncExecutionRequestRef = requestRef.asInstanceOf[AsyncExecutionRequestRef] + val nodeAction = new DataCheckerExecutionAction() + nodeAction.setId(UUID.randomUUID().toString()) + import scala.collection.JavaConversions.mapAsScalaMap + val InstanceConfig = this.service.getAppInstance.getConfig + val runTimeParams: scala.collection.mutable.Map[String, Object] = asyncExecutionRequestRef.getExecutionRequestRefContext().getRuntimeMap() + val variableParams: scala.collection.mutable.Map[String, Object]= asyncExecutionRequestRef.getJobContent.get("variable"). asInstanceOf[java.util.Map[String,Object]] + val inputParams =runTimeParams++variableParams + val properties = new Properties() + InstanceConfig.foreach { + case (key: String, value: Object) => + //避免密码被打印 + properties.put(key, value.toString) + } + val tmpProperties = new Properties() + runTimeParams.foreach( + record=> + if (null == record._2) { + properties.put(record._1, "") + }else { + if (record._1.equalsIgnoreCase("job.desc")) { + val rows = record._2.asInstanceOf[String].split("\n") + rows.foreach(row => if (row.contains("=")) { + val endLocation = row.indexOf("="); + val rowKey = row.substring(0, endLocation) + val rowEnd = row.substring(endLocation + 1) + tmpProperties.put(rowKey, rowEnd) + }) + } else { + tmpProperties.put(record._1, record._2) + } + } + ) + tmpProperties.foreach { record => + logger.info("request params key : " + record._1 + ",value : " + record._2) + if (null == record._2) { + properties.put(record._1, "") + } + else { + if(inputParams.exists(x=>x._1.equalsIgnoreCase(VariableUtils.RUN_DATE))) { + val tmp:util.HashMap[String, Any] = new util.HashMap[String,Any]() + tmp.put(VariableUtils.RUN_DATE,inputParams.get(VariableUtils.RUN_DATE).getOrElse(null)) + properties.put(record._1,VariableUtils.replace(record._2.toString,tmp)) + }else { + properties.put(record._1, VariableUtils.replace(record._2.toString)) + } + } + } + Utils.tryCatch({ + val dc = new DataChecker(properties, nodeAction) + dc.run() + nodeAction.setDc(dc) + })(t => { + logger.error("DataChecker run failed for " + t.getMessage, t) + putErrorMsg("DataChecker run failed! " + t.getMessage, t, nodeAction) + }) + nodeAction + + } + + override def state(action: RefExecutionAction): RefExecutionState = { + action match { + case action: DataCheckerExecutionAction => { + action.getExecutionRequestRefContext.appendLog("DataCheck is running!") + if (action.state.isCompleted) return action.state + Utils.tryCatch(action.dc.begineCheck(action))(t => { + action.setState(RefExecutionState.Failed) + logger.error("DataChecker run failed for " + t.getMessage, t) + putErrorMsg("DataChecker run failed! " + t.getMessage, t, action) + }) + action.state + } + case _ => RefExecutionState.Failed + } + } + + override def result(action: RefExecutionAction): CompletedExecutionResponseRef = { + val response:DataCheckerCompletedExecutionResponseRef = new DataCheckerCompletedExecutionResponseRef(200) + action match { + case action: DataCheckerExecutionAction => { + if (action.state.equals(RefExecutionState.Success)) { + response.setIsSucceed(true) + } else { + response.setErrorMsg(action.response.getErrorMsg) + response.setIsSucceed(false) + } + response + } + case _ => { + response.setIsSucceed(false) + response + } + + + } + } + + override def kill(action: RefExecutionAction): Boolean = action match { + case longTermAction: DataCheckerExecutionAction => + longTermAction.setKilledFlag(true) + longTermAction.setState(RefExecutionState.Killed) + true + } + + override def progress(action: RefExecutionAction): Float = { + //todo complete progress + 0.5f + } + + override def log(action: RefExecutionAction): String = { + action match { + case action: DataCheckerExecutionAction => { + if (!action.state.isCompleted) { + LogUtils.generateInfo("DataChecker is waiting for tables") + } else { + LogUtils.generateInfo("DataChecker successfully received info of tables") + } + } + case _ => LogUtils.generateERROR("Error for NodeExecutionAction ") + } + + } + + override def createAsyncResponseRef(requestRef: ExecutionRequestRef, action: RefExecutionAction): AsyncExecutionResponseRef = { + action match { + case action: DataCheckerExecutionAction => { + val response = super.createAsyncResponseRef(requestRef,action) + response.setMaxLoopTime(action.dc.maxWaitTime) + response.setAskStatePeriod(action.dc.queryFrequency) + response + } + } + } + + + override def setDevelopmentService(service: DevelopmentService): Unit = { + this.service = service + } +} diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/pom.xml b/dss-appconn/appconns/dss-eventchecker-appconn/pom.xml new file mode 100644 index 000000000..1c5808677 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/pom.xml @@ -0,0 +1,171 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-eventchecker-appconn + + + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-development-process-standard-execution + ${dss.version} + + + + org.apache.commons + commons-lang3 + 3.4 + + + + com.alibaba + druid + 1.0.28 + + + + log4j + log4j + 1.2.17 + + + + com.webank.wedatasphere.linkis + linkis-cs-client + ${linkis.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + + + + src/main/resources + + **/*.properties + **/application.yml + **/bootstrap.yml + **/log4j2.xml + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/assembly/distribution.xml new file mode 100644 index 000000000..a35e2cb15 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/assembly/distribution.xml @@ -0,0 +1,66 @@ + + + + dss-eventchecker-appconn + + dir + + true + eventchecker + + + + + + lib + true + true + false + true + true + + + + + + ${basedir}/src/main/resources + + appconn.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + log4j.properties + log4j2.xml + + 0777 + conf + unix + + + + + + diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/EventCheckerAppConn.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/EventCheckerAppConn.java new file mode 100644 index 000000000..bd277ca3e --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/EventCheckerAppConn.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker; + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractAppConn; +import com.webank.wedatasphere.dss.appconn.eventchecker.standard.EventCheckerDevelopmentStandard; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; + +public class EventCheckerAppConn extends AbstractAppConn implements OnlyDevelopmentAppConn { + + private EventCheckerDevelopmentStandard standard; + + @Override + protected void initialize() { + standard = new EventCheckerDevelopmentStandard(); + } + + @Override + public DevelopmentIntegrationStandard getOrCreateDevelopmentStandard() { + return standard; + } + +} diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/EventCheckerCompletedExecutionResponseRef.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/EventCheckerCompletedExecutionResponseRef.java new file mode 100644 index 000000000..603766fbd --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/EventCheckerCompletedExecutionResponseRef.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker; + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; + +import java.util.Map; + +public class EventCheckerCompletedExecutionResponseRef extends CompletedExecutionResponseRef { + private Exception exception; + public void setException(Exception exception) { + this.exception = exception; + } + + @Override + public Exception getException() { + return exception; + } + + public EventCheckerCompletedExecutionResponseRef(int status) { + super(status); + } + + public EventCheckerCompletedExecutionResponseRef(String responseBody, int status) { + super(responseBody, status); + } + + public void setStatus(int status){ + this.status = status; + } + + @Override + public Map toMap() { + return null; + } +} diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/adapter/EventCheckAdapter.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/adapter/EventCheckAdapter.java new file mode 100644 index 000000000..f7004bb38 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/adapter/EventCheckAdapter.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.adapter; + +import org.apache.log4j.Logger; + +import java.util.Properties; + +public interface EventCheckAdapter { + + boolean sendMsg(int jobId, Properties props, Logger log); + + boolean reciveMsg(int jobId, Properties props, Logger log); + +} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/EventDruidFactory.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/connector/EventDruidFactory.java similarity index 87% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/EventDruidFactory.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/connector/EventDruidFactory.java index fb1e3060d..3f0f0650b 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/connector/EventDruidFactory.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/connector/EventDruidFactory.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,7 +14,7 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.connector; +package com.webank.wedatasphere.dss.appconn.eventchecker.connector; import com.alibaba.druid.pool.DruidDataSource; @@ -25,13 +24,6 @@ import java.util.Base64; import java.util.Properties; -/** - * @author georgeqiao - * @Title: EventDruidFactory - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ public class EventDruidFactory { private static DruidDataSource msgInstance; @@ -55,14 +47,19 @@ private static DruidDataSource createDataSource(Properties props, Logger log, St String url = null; String username = null; String password = null; + String loginType = null; if(type.equals("Msg")){ name = props.getProperty("msg.eventchecker.jdo.option.name"); url = props.getProperty("msg.eventchecker.jdo.option.url"); username = props.getProperty("msg.eventchecker.jdo.option.username"); + loginType = props.getProperty("msg.eventchecker.jdo.option.login.type"); try { -// password = new String(Base64.getDecoder().decode(props.getProperty("msg.eventchecker.jdo.option.password").getBytes()),"UTF-8"); - password = props.getProperty("msg.eventchecker.jdo.option.password"); + if("base64".equals(loginType)) { + password = new String(Base64.getDecoder().decode(props.getProperty("msg.eventchecker.jdo.option.password").getBytes()), "UTF-8"); + }else{ + password = props.getProperty("msg.eventchecker.jdo.option.password"); + } } catch (Exception e){ log.error("password decore failed" + e); } diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/cs/CSEventReceiverHelper.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/cs/CSEventReceiverHelper.java new file mode 100644 index 000000000..74fea4ff2 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/cs/CSEventReceiverHelper.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.cs; + +import com.google.gson.Gson; +import com.webank.wedatasphere.linkis.cs.client.service.CSVariableService; +import com.webank.wedatasphere.linkis.cs.client.utils.ContextServiceUtils; +import com.webank.wedatasphere.linkis.cs.client.utils.SerializeHelper; +import com.webank.wedatasphere.linkis.cs.common.entity.enumeration.ContextScope; +import com.webank.wedatasphere.linkis.cs.common.entity.enumeration.ContextType; +import com.webank.wedatasphere.linkis.cs.common.entity.object.LinkisVariable; +import com.webank.wedatasphere.linkis.cs.common.entity.source.CommonContextKey; +import com.webank.wedatasphere.linkis.cs.common.entity.source.ContextKey; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Properties; + +public class CSEventReceiverHelper { + + private static final Logger LOGGER = LoggerFactory.getLogger(CSEventReceiverHelper.class); + + private static Gson gson = new Gson(); + + public static void putVariable(Properties properties, String msgBody, String saveKey) { + String contextIDStr = ContextServiceUtils.getContextIDStrByProperties(properties); + String nodeNameStr = ContextServiceUtils.getNodeNameStrByProperties(properties); + try { + + String key = saveKey; + String value = msgBody; + ContextKey contextKey = new CommonContextKey(); + contextKey.setContextScope(ContextScope.PUBLIC); + contextKey.setContextType(ContextType.OBJECT); + contextKey.setKey(CSCommonUtils.getVariableKey(nodeNameStr, key)); + LinkisVariable varValue = new LinkisVariable(); + varValue.setKey(key); + varValue.setValue(value); + CSVariableService.getInstance().putVariable(contextIDStr, SerializeHelper.serializeContextKey(contextKey), varValue); + } catch (Exception e) { + LOGGER.error("Failed to put variable to cs", e); + } + } +} diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/entity/EventChecker.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/entity/EventChecker.java new file mode 100644 index 000000000..e3ddf7a47 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/entity/EventChecker.java @@ -0,0 +1,227 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.entity; + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.appconn.eventchecker.cs.CSEventReceiverHelper; +import com.webank.wedatasphere.dss.appconn.eventchecker.execution.EventCheckerExecutionAction; +import com.webank.wedatasphere.dss.appconn.eventchecker.service.EventCheckerService; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionState; +import org.apache.commons.lang3.StringUtils; +import org.apache.log4j.Logger; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +public class EventChecker implements Runnable{ + public final static String WAIT_TIME = "max.receive.hours"; + public final static String WAIT_FOR_TIME = "wait.for.time"; + public final static String QUERY_FREQUENCY = "query.frequency"; + public final static String MSGTYPE="msg.type"; + public final static String SENDER="msg.sender"; + public final static String RECEIVER="msg.receiver"; + public final static String TOPIC="msg.topic"; + public final static String MSGNAME="msg.name"; + public final static String MSG="msg.body"; + public final static String EXEC_ID = "azkaban.flow.execid"; + public final static String SAVE_KEY="msg.savekey"; + public final static String USER_TIME="msg.init.querytime"; + public final static String TODAY="only.receive.today"; + public final static String AFTERSEND="msg.after.send"; + + private Properties p; + private String jobId; + private int execId; + private EventCheckerService wbDao=null; + EventCheckerExecutionAction backAction = null; + public Long maxWaitTime; + public int queryFrequency; + + private static Pattern pattern = Pattern.compile("[a-zA-Z_0-9@\\-]+"); + + private static final Logger logger = Logger.getRootLogger(); + + public EventChecker(Properties p, EventCheckerExecutionAction action) { + this.p = p; + this.jobId = "1"; + backAction = action; + String waitTime = p.getProperty(EventChecker.WAIT_TIME, "1"); + Double doubleWaitTime = Double.valueOf(waitTime) * 3600 * 1000; + maxWaitTime = Long.valueOf(doubleWaitTime.longValue()); + String query_frequency = p.getProperty(EventChecker.QUERY_FREQUENCY, "30000"); + queryFrequency = Integer.valueOf(query_frequency); + if(queryFrequency <10000){ + queryFrequency = 10000; + } + } + + @Override + public void run() { + try { + backAction.setState(RefExecutionState.Running); + if (p == null) { + throw new RuntimeException("Properties is null. Can't continue"); + } + if (checkParamMap(p, MSGTYPE)) { + throw new RuntimeException("parameter " + MSGTYPE + " can not be blank."); + } + if (checkParamMap(p, TOPIC)) { + throw new RuntimeException("parameter " + TOPIC + " can not be blank."); + } else { + String topic = p.getProperty(TOPIC); + if (!topic.matches("[^_]*_[^_]*_[^_]*")) { + throw new RuntimeException("Error format of topic parameter. Accept: XX_XX_XX."); + } + } + if (checkParamMap(p, MSGNAME)) { + throw new RuntimeException("parameter " + MSGNAME + " can not be blank."); + } + wbDao = EventCheckerService.getInstance(); + execId = Integer.parseInt(jobId); + boolean success = false; + if (p.getProperty(MSGTYPE).equals("SEND")) { + if (checkParamMap(p, SENDER)) { + throw new RuntimeException("parameter " + SENDER + " can not be blank."); + } else { + String sender = p.getProperty(SENDER); + if (!sender.matches("[^@]*@[^@]*@[^@]*")) { + throw new RuntimeException("Error format of sender parameter. Accept: XX@XX@XX."); + } + } + if (p.containsKey(MSG) && StringUtils.isNotEmpty(p.getProperty(MSG)) && p.getProperty(MSG).length() > 250) { + throw new RuntimeException("parameter " + MSG + " length less than 250 !"); + } + success = wbDao.sendMsg(execId, p, logger); + if (success) { + backAction.setState(RefExecutionState.Success); + + } else { + throw new RuntimeException("Failed Send message."); + } + }else if(p.getProperty(MSGTYPE).equals("RECEIVE")) { + backAction.eventType("RECEIVE"); + receiveMsg(); + } else + { + throw new RuntimeException("Please input correct parameter of msg.type, Select RECEIVE Or SEND."); + } + }catch (Exception ex){ + backAction.setState(RefExecutionState.Failed); + throw ex; + } + + } + + public boolean receiveMsg(){ + boolean success = false; + if(p.getProperty(MSGTYPE).equals("RECEIVE")) { + if (checkParamMap(p, RECEIVER)) { + backAction.setState(RefExecutionState.Failed); + throw new RuntimeException("parameter " + RECEIVER + " can not be blank."); + } else { + String receiver = p.getProperty(RECEIVER); + if (!receiver.matches("[^@]*@[^@]*@[^@]*")) { + backAction.setState(RefExecutionState.Failed); + throw new RuntimeException("Error format of receiver parameter. Accept: XX@XX@XX."); + } + } + String userTime = checkTimeParamMap(p, USER_TIME); + if (StringUtils.isNotEmpty(userTime)) { + p.put(USER_TIME, userTime); + } + success = wbDao.reciveMsg(execId, p, logger); + if (success) { + backAction.saveKeyAndValue(getJobSaveKeyAndValue()); + backAction.setState(RefExecutionState.Success); + } else { + backAction.setState(RefExecutionState.Running); + } + } + return success; + } + + public String getJobSaveKeyAndValue(){ + Map saveValueMap = new HashMap<>(); + String msgBody = p.getProperty(MSG, "{}"); + String saveKey = p.getProperty(SAVE_KEY,"msg.body"); + CSEventReceiverHelper.putVariable(this.p, msgBody, saveKey); + if(StringUtils.isEmpty(saveKey)){ + saveValueMap.put("msg.body", msgBody); + }else { + saveValueMap.put(saveKey, msgBody); + } + Gson gson = new Gson(); + String saveValueJson = gson.toJson(saveValueMap); + logger.info("Output msg body: "+saveValueJson); + return saveValueJson; + } + + public void cancel() throws InterruptedException { + } + + private boolean checkParamMap(Properties p, String key){ + boolean checkFlag = false; + if(!p.containsKey(key)){ + throw new RuntimeException("parameter " + key + " is Empty."); + } + if(p.containsKey(key)){ + if(StringUtils.isEmpty(p.getProperty(key))){ + checkFlag = true; + } + } + if(!MSG.equals(key) && StringUtils.contains(p.getProperty(key), " ")){ + throw new RuntimeException("parameter " + key + " can not contains space !"); + } + if(!checkNoStandardStr(p.getProperty(key))){ + throw new RuntimeException("parameter " + key + " Accept letter and number and _@- only."); + } + if(p.getProperty(key).length() > 45){ + throw new RuntimeException("parameter " + key + " length less than 45 !"); + } + return checkFlag; + } + + private boolean checkNoStandardStr(String param){ + Matcher matcher = pattern.matcher(param); + return matcher.matches(); + } + + private void checkTimeParam(Properties p, String key){ + if(p.containsKey(key)){ + String waitForTime= p.getProperty(key); + if(!waitForTime.matches("^(0?[0-9]|1[0-9]|2[0-3]):(0?[0-9]|[1-5][0-9])$")){ + throw new RuntimeException("Parameter " + key + " Time format error ! For example: HH:mm"); + } + } + } + + private String checkTimeParamMap(Properties p, String key){ + if(p.containsKey(key)){ + String userTime = p.getProperty(key); + Pattern ptime = Pattern.compile("^([1][7-9][0-9][0-9]|[2][0][0-9][0-9])(\\-)([0][1-9]|[1][0-2])(\\-)([0-2][1-9]|[3][0-1])(\\s)([0-1][0-9]|[2][0-3])(:)([0-5][0-9])(:)([0-5][0-9])$"); + Matcher m = ptime.matcher(userTime); + if(!m.matches()){ + throw new RuntimeException("Parameter " + key + " Time format error ! For example: yyyy-MM-dd HH:mm:ss"); + } + return userTime; + }else{ + return null; + } + } +} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/exception/UndefinedPropertyException.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/exception/UndefinedPropertyException.java similarity index 84% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/exception/UndefinedPropertyException.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/exception/UndefinedPropertyException.java index d92be51c3..fe86138ea 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/exception/UndefinedPropertyException.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/exception/UndefinedPropertyException.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,7 +14,7 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.exception; +package com.webank.wedatasphere.dss.appconn.eventchecker.exception; /** * Indicates that a required property is missing from the Props diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/AbstractEventCheck.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/AbstractEventCheck.java new file mode 100644 index 000000000..aa63a87f7 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/AbstractEventCheck.java @@ -0,0 +1,161 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.service; + +import com.alibaba.druid.pool.DruidDataSource; +import com.webank.wedatasphere.dss.appconn.eventchecker.connector.EventDruidFactory; +import com.webank.wedatasphere.dss.appconn.eventchecker.adapter.EventCheckAdapter; +import com.webank.wedatasphere.dss.appconn.eventchecker.entity.EventChecker; + +import org.apache.log4j.Logger; + +import java.net.InetAddress; +import java.net.NetworkInterface; +import java.net.SocketException; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.Enumeration; +import java.util.Properties; + +import javax.sql.DataSource; + +public abstract class AbstractEventCheck implements EventCheckAdapter { + static DataSource msgDS; + String topic; + String msgName; + String receiver; + String sender; + String receiveToday; + String userTime; + String waitTime; + String query_frequency; + String wait_for_time; + String msg; + String afterSend; + + DataSource getMsgDS(Properties props, Logger log){ + if (msgDS == null) { + msgDS = EventDruidFactory.getMsgInstance(props, log); + if (msgDS == null) { + log.error("Error getting Druid DataSource instance"); + } + } + return msgDS; + } + + void initECParams(Properties props){ + topic = props.getProperty(EventChecker.TOPIC); + msgName = props.getProperty(EventChecker.MSGNAME); + receiver = props.getProperty(EventChecker.RECEIVER); + sender = props.getProperty(EventChecker.SENDER); + msg = props.getProperty(EventChecker.MSG); + receiveToday = props.getProperty(EventChecker.TODAY); + userTime = props.getProperty(EventChecker.USER_TIME); + waitTime = props.getProperty(EventChecker.WAIT_TIME, "1"); + query_frequency = props.getProperty(EventChecker.QUERY_FREQUENCY, "30000"); + afterSend = props.getProperty(EventChecker.AFTERSEND); + } + + Connection getEventCheckerConnection(Properties props, Logger log){ + Connection connection = null; + try { + connection = getMsgDS(props,log).getConnection(); + } catch (SQLException e) { + throw new RuntimeException("Error getting DB Connection instance {} " + e); + } + return connection; + } + + @Override + public boolean sendMsg(int jobId, Properties props, Logger log) { + return false; + } + + @Override + public boolean reciveMsg(int jobId, Properties props, Logger log) { + return false; + } + + void closeConnection(Connection conn, Logger log) { + if (conn != null) { + try { + conn.close(); + } catch (SQLException e) { + log.error("Error closing connection", e); + } + } + } + + void closeQueryRef(ResultSet rs, Logger log) { + if (rs != null) { + try { + rs.close(); + } catch (SQLException e) { + log.error("Error closing result set", e); + } + } + + } + + void closeQueryStmt(PreparedStatement stmt, Logger log) { + if (stmt != null) { + try { + stmt.close(); + } catch (SQLException e) { + log.error("Error closing result stmt", e); + } + } + + } + + + public static void closeDruidDataSource() { + DruidDataSource msgDSObject = (DruidDataSource) msgDS; + if (msgDSObject != null) { + msgDSObject.close(); + } + + } + + String getLinuxLocalIp(Logger log) { + String ip = "127.0.0.1"; + try { + for (Enumeration en = NetworkInterface.getNetworkInterfaces(); en.hasMoreElements(); ) { + NetworkInterface intf = en.nextElement(); + String name = intf.getName(); + if (!name.contains("docker") && !name.contains("lo")) { + for (Enumeration enumIpAddr = intf.getInetAddresses(); enumIpAddr.hasMoreElements(); ) { + InetAddress inetAddress = enumIpAddr.nextElement(); + if (!inetAddress.isLoopbackAddress()) { + String ipaddress = inetAddress.getHostAddress().toString(); + if (!ipaddress.contains("::") && !ipaddress.contains("0:0:") && !ipaddress.contains("fe80")) { + ip = ipaddress; + } + } + } + } + } + } catch (SocketException ex) { + log.warn("get ip failed", ex); + + } + log.info("Send IP:" + ip); + return ip; + } +} diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/AbstractEventCheckReceiver.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/AbstractEventCheckReceiver.java new file mode 100644 index 000000000..dd48920fd --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/AbstractEventCheckReceiver.java @@ -0,0 +1,176 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.service; + +import com.webank.wedatasphere.dss.appconn.eventchecker.entity.EventChecker; + +import org.apache.commons.lang3.StringUtils; +import org.apache.commons.lang3.time.DateFormatUtils; +import org.apache.log4j.Logger; + +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.Date; +import java.util.Properties; + +public class AbstractEventCheckReceiver extends AbstractEventCheck{ + /** + * Fill the result into the source + */ + String setConsumedMsg(Properties props, Logger log, String[] consumedMsgInfo){ + String vNewMsgID = ""; + try { + if(consumedMsgInfo!=null && consumedMsgInfo.length == 4){ + vNewMsgID = consumedMsgInfo[0]; + String vMsgName = consumedMsgInfo[1]; + String vSender = consumedMsgInfo[2]; + String vMsg = consumedMsgInfo[3]; + if (null == vMsg) { + props.put(EventChecker.MSG, "NULL"); + } else { + props.put(EventChecker.MSG, vMsg); + } + log.info("Received message : messageID: " + vNewMsgID + ", messageName: " + vMsgName + ", receiver: " + vSender + + ", messageBody: " + vMsg); + } + }catch (Exception e) { + log.error("Error set consumed message failed {} setConsumedMsg failed" + e); + return vNewMsgID; + } + return vNewMsgID; + } + + /** + * Update consumption status + */ + boolean updateMsgOffset(int jobId, Properties props, Logger log, String[] consumedMsgInfo,String lastMsgId){ + boolean result = false; + String vNewMsgID = "-1"; + PreparedStatement updatePstmt = null; + Connection msgConn = null; + vNewMsgID = setConsumedMsg(props,log,consumedMsgInfo); + try { + if(StringUtils.isNotEmpty(vNewMsgID) && StringUtils.isNotBlank(vNewMsgID) && !"-1".equals(vNewMsgID)){ + msgConn = getEventCheckerConnection(props,log); + if(msgConn == null) return false; + int vProcessID = jobId; + String vReceiveTime = DateFormatUtils.format(new Date(), "yyyy-MM-dd HH:mm:ss");; + String sqlForUpdateMsg = "INSERT INTO event_status(receiver,topic,msg_name,receive_time,msg_id) VALUES(?,?,?,?,?) ON DUPLICATE KEY UPDATE receive_time=VALUES(receive_time),msg_id= CASE WHEN msg_id= " + lastMsgId + " THEN VALUES(msg_id) ELSE msg_id END"; + log.info("last message offset {} is:" + lastMsgId); + updatePstmt = msgConn.prepareCall(sqlForUpdateMsg); + updatePstmt.setString(1, receiver); + updatePstmt.setString(2, topic); + updatePstmt.setString(3, msgName); + updatePstmt.setString(4, vReceiveTime); + updatePstmt.setString(5, vNewMsgID); + int updaters = updatePstmt.executeUpdate(); + log.info("updateMsgOffset successful {} update result is:" + updaters); + if(updaters != 0){ + log.info("Received message successfully , update message status succeeded, consumed flow execution ID: " + vProcessID); + //return true after update success + result = true; + }else{ + log.info("Received message successfully , update message status failed, consumed flow execution ID: " + vProcessID); + result = false; + } + }else{ + result = false; + } + }catch (SQLException e){ + log.error("Error update Msg Offset" + e); + return false; + }finally { + closeQueryStmt(updatePstmt, log); + closeConnection(msgConn, log); + } + return result; + } + + /** + * get consumption progress + */ + String getOffset(int jobId, Properties props, Logger log){ + String sqlForReadMsgID = "SELECT msg_id FROM event_status WHERE receiver=? AND topic=? AND msg_name=?"; + PreparedStatement pstmtForGetID = null; + Connection msgConn = null; + ResultSet rs = null; + boolean flag = false; + String lastMsgId = "0"; + try { + msgConn = getEventCheckerConnection(props,log); + pstmtForGetID = msgConn.prepareCall(sqlForReadMsgID); + pstmtForGetID.setString(1, receiver); + pstmtForGetID.setString(2, topic); + pstmtForGetID.setString(3, msgName); + rs = pstmtForGetID.executeQuery(); + lastMsgId = rs.last()==true ? rs.getString("msg_id"):"0"; + } catch (SQLException e) { + throw new RuntimeException("get Offset failed " + e); + }finally { + closeQueryStmt(pstmtForGetID,log); + closeConnection(msgConn,log); + closeQueryRef(rs,log); + } + log.info("The last record id was " + lastMsgId); + return lastMsgId; + } + + /** + * Consistent entrance to consumer message + */ + String[] getMsg(Properties props, Logger log,String ... params){ + String sqlForReadTMsg = "SELECT * FROM event_queue WHERE topic=? AND msg_name=? AND send_time >=? AND send_time <=? AND msg_id >? ORDER BY msg_id ASC LIMIT 1"; + PreparedStatement pstmt = null; + Connection msgConn = null; + ResultSet rs = null; + String[] consumedMsgInfo = null; + try { + msgConn = getEventCheckerConnection(props,log); + pstmt = msgConn.prepareCall(sqlForReadTMsg); + pstmt.setString(1, topic); + pstmt.setString(2, msgName); + pstmt.setString(3, params[0]); + pstmt.setString(4, params[1]); + pstmt.setString(5, params[2]); + log.info("param {} StartTime: " + params[0] + ", EndTime: " + params[1] + + ", Topic: " + topic + ", MessageName: " + msgName + ", LastMessageID: " + params[2]); + rs = pstmt.executeQuery(); + + if(rs.last()){ + consumedMsgInfo = new String[4]; + String[] msgKey = new String[]{"msg_id","msg_name","sender","msg"}; + for (int i = 0;i <= 3;i++) { + consumedMsgInfo[i] = rs.getString(msgKey[i]); + } + } + } catch (SQLException e) { + throw new RuntimeException("EventChecker failed to receive message" + e); + } finally { + closeQueryStmt(pstmt, log); + closeConnection(msgConn, log); + closeQueryRef(rs, log); + } + return consumedMsgInfo; + } + + @Override + public boolean reciveMsg(int jobId, Properties props, Logger log) { + return super.reciveMsg(jobId, props, log); + } +} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/DefaultEventcheckReceiver.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/DefaultEventcheckReceiver.java similarity index 93% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/DefaultEventcheckReceiver.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/DefaultEventcheckReceiver.java index 2759256b9..9fa17274f 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/DefaultEventcheckReceiver.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/DefaultEventcheckReceiver.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,11 +14,10 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service; +package com.webank.wedatasphere.dss.appconn.eventchecker.service; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.EventChecker; import org.apache.commons.lang3.time.DateFormatUtils; import org.apache.log4j.Logger; @@ -29,13 +27,6 @@ import java.util.Date; import java.util.Properties; -/** - * @author georgeqiao - * @Title: DefaultEventcheckReceiver - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ public class DefaultEventcheckReceiver extends AbstractEventCheckReceiver { String todayStartTime; String todayEndTime; diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/EventCheckSender.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckSender.java similarity index 89% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/EventCheckSender.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckSender.java index 662234ec4..8e21ab72d 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/EventCheckSender.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckSender.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,7 +14,7 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service; +package com.webank.wedatasphere.dss.appconn.eventchecker.service; import org.apache.commons.lang3.time.DateFormatUtils; import org.apache.log4j.Logger; @@ -26,13 +25,6 @@ import java.util.Date; import java.util.Properties; -/** - * @author georgeqiao - * @Title: EventCheckSender - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ public class EventCheckSender extends AbstractEventCheck { public EventCheckSender(Properties props) { diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckerExecuteService.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckerExecuteService.java new file mode 100644 index 000000000..0aeef73cb --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckerExecuteService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.service; + +import com.webank.wedatasphere.dss.appconn.eventchecker.execution.EventCheckerRefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExecutionService; + +public class EventCheckerExecuteService extends AbstractRefExecutionService { + + @Override + public RefExecutionOperation createRefExecutionOperation() { + EventCheckerRefExecutionOperation eventCheckerRefExecutionOperation = new EventCheckerRefExecutionOperation(); + eventCheckerRefExecutionOperation.setDevelopmentService(this); + return eventCheckerRefExecutionOperation; + } + +} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/EventCheckerService.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckerService.java similarity index 77% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/EventCheckerService.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckerService.java index 7f75170f5..2ca27bec3 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/EventCheckerService.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/service/EventCheckerService.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,19 +14,12 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service; +package com.webank.wedatasphere.dss.appconn.eventchecker.service; import org.apache.log4j.Logger; import java.util.Properties; -/** - * @author georgeqiao - * @Title: EventCheckerService - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ public class EventCheckerService { private static EventCheckerService instance; @@ -52,8 +44,6 @@ public boolean sendMsg(int jobId, Properties props, Logger log) { } /** - * 接收消息 接收消息先查询消费记录，有则从上一次消费后开始消费，没有则从任务启动时间点后开始消费。 - * 接收消息是以主动查询的方式进行的，在没有超出设定目标的时间内，反复查询目标消息。 * Receiving a message first queries the consumption record, * and then starts to consume after the last consumption, and no consumption * starts after the job starts. The received message is performed in an active diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/standard/EventCheckerDevelopmentStandard.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/standard/EventCheckerDevelopmentStandard.java new file mode 100644 index 000000000..597ea95e3 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/standard/EventCheckerDevelopmentStandard.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.standard; + +import com.webank.wedatasphere.dss.appconn.eventchecker.service.EventCheckerExecuteService; +import com.webank.wedatasphere.dss.standard.app.development.standard.OnlyExecutionDevelopmentStandard; +import com.webank.wedatasphere.dss.standard.app.development.service.RefExecutionService; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class EventCheckerDevelopmentStandard extends OnlyExecutionDevelopmentStandard { + + private static final Logger LOGGER = LoggerFactory.getLogger(EventCheckerDevelopmentStandard.class); + + @Override + protected RefExecutionService createRefExecutionService() { + return new EventCheckerExecuteService(); + } + + + @Override + public void init() { + LOGGER.info("class EventCheckerDevelopmentStandard init"); + } + + + @Override + public String getStandardName() { + return "EventCheckDevelopmentStandard"; + } + +} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/utils/Props.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/utils/Props.java similarity index 98% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/utils/Props.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/utils/Props.java index 1e19756a5..e4e8cc9b8 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/utils/Props.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/utils/Props.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,9 +14,9 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.utils; +package com.webank.wedatasphere.dss.appconn.eventchecker.utils; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.exception.UndefinedPropertyException; +import com.webank.wedatasphere.dss.appconn.eventchecker.exception.UndefinedPropertyException; import org.apache.log4j.Logger; @@ -163,11 +162,6 @@ public static Props clone(final Props p) { return copyNext(p); } - /** - * - * @param source - * @return - */ private static Props copyNext(final Props source) { Props priorNodeCopy = null; if (source.getParent() != null) { @@ -181,14 +175,9 @@ private static Props copyNext(final Props source) { return dest; } - /** - * - * @param inputStream - * @throws IOException - */ private void loadFrom(final InputStream inputStream) throws IOException { final Properties properties = new Properties(); - //解决.job文件中包含中文，读取乱码的问题。 + // Solve the problem that the. Job file contains Chinese and reads garbled code. BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream , "UTF-8")); properties.load(bufferedReader); this.put(properties); @@ -793,8 +782,6 @@ public void logProperties(final Logger logger, final String comment) { } } - /** - */ @Override public boolean equals(final Object o) { if (o == this) { @@ -827,9 +814,6 @@ public boolean equalsProps(final Props p) { return myKeySet.size() == p.getKeySet().size(); } - /** - * - */ @Override public int hashCode() { int code = this._current.hashCode(); @@ -839,9 +823,6 @@ public int hashCode() { return code; } - /** - * - */ @Override public String toString() { final StringBuilder builder = new StringBuilder("{"); diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/utils/Utils.java b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/utils/Utils.java similarity index 86% rename from eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/utils/Utils.java rename to dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/utils/Utils.java index 682033e55..a53dbf27e 100644 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/utils/Utils.java +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/eventchecker/utils/Utils.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,7 +14,7 @@ * */ -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.utils; +package com.webank.wedatasphere.dss.appconn.eventchecker.utils; /** * A util helper class full of static methods that are commonly used. diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/appjoint.properties b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/appjoint.properties new file mode 100644 index 000000000..9538027d4 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/appjoint.properties @@ -0,0 +1,24 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +msg.eventchecker.jdo.option.name=msg +msg.eventchecker.jdo.option.url=jdbc:mysql://127.0..0.1:3306/ +msg.eventchecker.jdo.option.username=user +msg.eventchecker.jdo.option.password= +msg.eventchecker.jdo.option.login.type=base64 + + + diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/log4j.properties b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/log4j.properties new file mode 100644 index 000000000..ee8619595 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/log4j2.xml b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/log4j2.xml new file mode 100644 index 000000000..8c40a73e8 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/resources/log4j2.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/eventchecker/execution/EventCheckerExecutionAction.scala b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/eventchecker/execution/EventCheckerExecutionAction.scala new file mode 100644 index 000000000..e449f6453 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/eventchecker/execution/EventCheckerExecutionAction.scala @@ -0,0 +1,63 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.execution + +import com.webank.wedatasphere.dss.appconn.eventchecker.EventCheckerCompletedExecutionResponseRef +import com.webank.wedatasphere.dss.appconn.eventchecker.entity.EventChecker +import com.webank.wedatasphere.dss.standard.app.development.listener.common.{AbstractRefExecutionAction, LongTermRefExecutionAction, RefExecutionState} + +class EventCheckerExecutionAction extends AbstractRefExecutionAction with LongTermRefExecutionAction { + private[this] var _state: RefExecutionState = _ + private var schedulerId: Int = _ + + def state: RefExecutionState = _state + + def setState(value: RefExecutionState): Unit = { + _state = value + } + + + val response = new EventCheckerCompletedExecutionResponseRef(200) + + private[this] var _saveKeyAndValue: String = null + + def saveKeyAndValue: String = _saveKeyAndValue + + def saveKeyAndValue(value: String): Unit = { + _saveKeyAndValue = value + } + + private[this] var _eventType: String = "SEND" + + def eventType: String = _eventType + + def eventType(value: String): Unit = { + _eventType = value + } + + private[this] var _ec: EventChecker = null + + def ec: EventChecker = _ec + + def setEc(value: EventChecker): Unit = { + _ec = value + } + + override def setSchedulerId(schedulerId: Int): Unit = this.schedulerId = schedulerId + + override def getSchedulerId: Int = schedulerId +} diff --git a/dss-appconn/appconns/dss-eventchecker-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/eventchecker/execution/EventCheckerRefExecutionOperation.scala b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/eventchecker/execution/EventCheckerRefExecutionOperation.scala new file mode 100644 index 000000000..bfd8a2593 --- /dev/null +++ b/dss-appconn/appconns/dss-eventchecker-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/eventchecker/execution/EventCheckerRefExecutionOperation.scala @@ -0,0 +1,178 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.eventchecker.execution + + + + +import java.util.{Properties, UUID} + +import com.webank.wedatasphere.dss.appconn.eventchecker.EventCheckerCompletedExecutionResponseRef +import com.webank.wedatasphere.dss.appconn.eventchecker.entity.EventChecker +import com.webank.wedatasphere.dss.standard.app.development.listener.{ExecutionLogListener, ExecutionResultListener} +import com.webank.wedatasphere.dss.standard.app.development.listener.common.{AsyncExecutionRequestRef, AsyncExecutionResponseRef, CompletedExecutionResponseRef, RefExecutionAction, RefExecutionState} +import com.webank.wedatasphere.dss.standard.app.development.listener.core.{Killable, LongTermRefExecutionOperation, Procedure} +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService +import com.webank.wedatasphere.linkis.common.log.LogUtils +import com.webank.wedatasphere.linkis.common.utils.Utils +import com.webank.wedatasphere.linkis.storage.LineRecord +import org.apache.commons.io.IOUtils +import org.slf4j.LoggerFactory; + + +class EventCheckerRefExecutionOperation extends LongTermRefExecutionOperation with Killable with Procedure{ + + + + private var service:DevelopmentService = _ + private val logger = LoggerFactory.getLogger(classOf[EventCheckerRefExecutionOperation]) + + + + + override def progress(action: RefExecutionAction): Float = { + //temp set + 0.5f + } + + override def log(action: RefExecutionAction): String = { + action match { + case action: EventCheckerExecutionAction => { + if (!action.state.isCompleted) { + LogUtils.generateInfo("EventChecker is sending or waiting for message") + } else { + LogUtils.generateInfo("EventChecker successfully received or send message") + } + } + case _ => LogUtils.generateERROR("Error NodeExecutionAction for log") + } + } + + override def kill(action: RefExecutionAction): Boolean = action match { + case longTermAction: EventCheckerExecutionAction => + longTermAction.setKilledFlag(true) + longTermAction.setState(RefExecutionState.Killed) + true + } + + protected def putErrorMsg(errorMsg: String, t: Throwable, action: EventCheckerExecutionAction): EventCheckerExecutionAction = t match { + + case t: Exception => + val response = action.response + response.setErrorMsg(errorMsg) + response.setException(t) + response.setIsSucceed(false) + action + } + + override def submit(requestRef: ExecutionRequestRef): RefExecutionAction = { + val asyncExecutionRequestRef = requestRef.asInstanceOf[AsyncExecutionRequestRef] + val nodeAction = new EventCheckerExecutionAction() + nodeAction.setId(UUID.randomUUID().toString()) + import scala.collection.JavaConversions.mapAsScalaMap + val InstanceConfig = this.service.getAppInstance.getConfig + val scalaParams: scala.collection.mutable.Map[String, Object] =asyncExecutionRequestRef.getExecutionRequestRefContext().getRuntimeMap() + val properties = new Properties() + InstanceConfig.foreach { record => + logger.info("request params key : " + record._1 + ",value : " + record._2) + if(null == record._2){ + properties.put(record._1, "")} + else { + properties.put(record._1, record._2.toString) + } + } + scalaParams.foreach { case (key, value) => + if (key != null && value != null) properties.put(key, value.toString) + } + Utils.tryCatch({ + val ec = new EventChecker(properties, nodeAction) + ec.run() + nodeAction.setEc(ec) + })(t => { + logger.error("EventChecker run failed for " + t.getMessage, t) + putErrorMsg("EventChecker run failed!" + t.getMessage, t, nodeAction) + }) + + nodeAction + } + + override def state(action: RefExecutionAction): RefExecutionState = { + action match { + case action: EventCheckerExecutionAction => { + action.getExecutionRequestRefContext.appendLog("EventCheck is running!") + if (action.state.isCompleted) return action.state + if (action.eventType.equals("RECEIVE")) { + Utils.tryCatch(action.ec.receiveMsg())(t => { + action.setState(RefExecutionState.Failed) + logger.error("EventChecker run failed for " + t.getMessage, t) + putErrorMsg("EventChecker run failed!" + t.getMessage, t, action) + false + }) + } + action.state + } + case _ => RefExecutionState.Failed + } + } + + override def result(action: RefExecutionAction): CompletedExecutionResponseRef = { + val response = new EventCheckerCompletedExecutionResponseRef(200) + action match { + case action: EventCheckerExecutionAction => { + if (action.state.equals(RefExecutionState.Success)) { + val resultSetWriter =action.getExecutionRequestRefContext.createTextResultSetWriter() + var resultStr = "EventChecker runs successfully!" + if (action.saveKeyAndValue != null) { + resultStr = action.saveKeyAndValue + logger.info("EventChecker save receive value: " + resultStr) + } + Utils.tryFinally { + resultSetWriter.addMetaData(null) + resultSetWriter.addRecord(new LineRecord(resultStr)) + }(IOUtils.closeQuietly(resultSetWriter)) + response.setIsSucceed(true) + } else { + response.setException(action.response.getException) + response.setIsSucceed(false) + } + response + } + case _ => + response.setIsSucceed(false); + response + } + + } + + override def createAsyncResponseRef(requestRef: ExecutionRequestRef, action: RefExecutionAction): AsyncExecutionResponseRef = { + action match { + case action: EventCheckerExecutionAction => { + val response = super.createAsyncResponseRef(requestRef,action) + response.setAction(action) + response.setMaxLoopTime(action.ec.maxWaitTime) + response.setAskStatePeriod(action.ec.queryFrequency) + response + } + } + } + + override def setDevelopmentService(service: DevelopmentService): Unit = { + this.service = service + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/pom.xml b/dss-appconn/appconns/dss-orchestrator-framework-appconn/pom.xml new file mode 100644 index 000000000..045016f3e --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/pom.xml @@ -0,0 +1,142 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator-framework-appconn + + + + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + + + com.webank.wedatasphere.linkis + linkis-module + + + com.webank.wedatasphere.linkis + linkis-rpc + + + com.webank.wedatasphere.dss + dss-contextservice + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-orchestrator-common + ${dss.version} + compile + + + com.webank.wedatasphere.dss + dss-workflow-common + ${dss.version} + compile + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-sender-service + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/assembly/distribution.xml new file mode 100644 index 000000000..dd3313ba8 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/assembly/distribution.xml @@ -0,0 +1,66 @@ + + + + dss-orchestrator-framework-appconn + + dir + + true + orchestrator-framework + + + + + + lib + true + true + false + true + true + + + + + + ${basedir}/src/main/resources + + appconn.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + log4j.properties + log4j2.xml + + 0777 + conf + unix + + + + + + diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/DefaultOrchestratorFrameworkAppConn.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/DefaultOrchestratorFrameworkAppConn.java new file mode 100644 index 000000000..10c123354 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/DefaultOrchestratorFrameworkAppConn.java @@ -0,0 +1,66 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator; + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractAppConn; +import com.webank.wedatasphere.dss.appconn.orchestrator.standard.OrchestratorFrameworkStandard; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorFrameworkAppConn; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppDesc; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; + +public class DefaultOrchestratorFrameworkAppConn extends AbstractAppConn implements OrchestratorFrameworkAppConn, OnlyDevelopmentAppConn { + + private static final Logger LOGGER = LoggerFactory.getLogger(DefaultOrchestratorFrameworkAppConn.class); + + private OrchestratorFrameworkStandard orchestratorFrameworkStandard; + private AppDesc appDesc; + + private final List standards = new ArrayList<>(); + + + @Override + public void setAppDesc(AppDesc appDesc) { + this.appDesc = appDesc; + } + + @Override + public List getAppStandards() { + return this.standards; + } + + @Override + protected void initialize() { + orchestratorFrameworkStandard = OrchestratorFrameworkStandard.getInstance(); + } + + @Override + public AppDesc getAppDesc() { + return this.appDesc; + } + + @Override + public DevelopmentIntegrationStandard getOrCreateDevelopmentStandard() { + return OrchestratorFrameworkStandard.getInstance(); + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/conf/OrchestratorConf.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/conf/OrchestratorConf.java new file mode 100644 index 000000000..a5d2b6098 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/conf/OrchestratorConf.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.conf; + +public class OrchestratorConf { + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkCreationOperation.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkCreationOperation.java new file mode 100644 index 000000000..6eb10fee6 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkCreationOperation.java @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.operation; + +import com.webank.wedatasphere.dss.appconn.orchestrator.ref.DefaultOrchestratorCreateResponseRef; +import com.webank.wedatasphere.dss.common.protocol.ResponseCreateOrchestrator; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestCreateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateResponseRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class OrchestratorFrameworkCreationOperation implements + RefCreationOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorFrameworkCreationOperation.class); + + private final Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(); + private DevelopmentService developmentService; + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + @Override + public OrchestratorCreateResponseRef createRef(OrchestratorCreateRequestRef requestRef) throws ExternalOperationFailedException { + if (null == requestRef) { + LOGGER.error("requestRef is null can not create Ref"); + return null; + } + RequestCreateOrchestrator createRequest = new RequestCreateOrchestrator(requestRef.getUserName(), + requestRef.getWorkspaceName(), requestRef.getProjectName(), + requestRef.getProjectId(), requestRef.getDSSOrchestratorInfo().getDesc(), + requestRef.getDSSOrchestratorInfo(), requestRef.getDSSLabels()); + ResponseCreateOrchestrator createResponse = null; + try { + createResponse = (ResponseCreateOrchestrator) sender.ask(createRequest); + } catch (Exception e) { + DSSExceptionUtils.dealErrorException(60015, "create orchestrator ref failed", e, + ExternalOperationFailedException.class); + } + if (createResponse == null) { + LOGGER.error("createResponse is null, can not get correct response"); + return null; + } + LOGGER.info("End to ask to create orchestrator, orcId is {} orcVersionId is {}", + createResponse.orchestratorId(), createResponse.orchestratorVersionId()); + DefaultOrchestratorCreateResponseRef createResponseRef = new DefaultOrchestratorCreateResponseRef(); + createResponseRef.setOrcId(createResponse.orchestratorId()); + createResponseRef.setOrchestratorVersionId(createResponse.orchestratorVersionId()); + return createResponseRef; + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkDeleteOperation.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkDeleteOperation.java new file mode 100644 index 000000000..30978e5a8 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkDeleteOperation.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.operation; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestDeleteOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorDeleteRequestRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class OrchestratorFrameworkDeleteOperation implements + RefDeletionOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorFrameworkDeleteOperation.class); + + @Override + public void setDevelopmentService(DevelopmentService service) { + } + + @Override + public void deleteRef(OrchestratorDeleteRequestRef requestRef) throws ExternalOperationFailedException { + LOGGER.info("Begin to ask to delete orchestrator, requestRef is {}", requestRef); + RequestDeleteOrchestrator deleteRequest = new RequestDeleteOrchestrator(requestRef.getUserName(), + requestRef.getWorkspaceName(), requestRef.getProjectName(), + requestRef.getOrcId(), requestRef.getDSSLabels()); + ResponseOperateOrchestrator deleteResponse = null; + try { + Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(requestRef.getDSSLabels()); + deleteResponse = (ResponseOperateOrchestrator) sender.ask(deleteRequest); + LOGGER.info("End to ask to delete orchestrator , responseRef is {}", deleteResponse); + if (deleteResponse == null || !deleteResponse.getJobStatus().equals(JobStatus.Success)){ + LOGGER.error("delete response is null or delete response status is not success"); + DSSExceptionUtils.dealErrorException(60075, "failed to delete ref", ExternalOperationFailedException.class); + } + } catch (Exception e) { + DSSExceptionUtils.dealErrorException(60015, "delete orchestrator ref failed", ExternalOperationFailedException.class); + } + } + +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkExportOperation.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkExportOperation.java new file mode 100644 index 000000000..d0c0fc29c --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkExportOperation.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.operation; + +import com.webank.wedatasphere.dss.appconn.orchestrator.ref.DefaultOrchestratorExportResponseRef; +import com.webank.wedatasphere.dss.common.protocol.ResponseExportOrchestrator; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestExportOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportResponseRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class OrchestratorFrameworkExportOperation implements RefExportOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorFrameworkExportOperation.class); + + private final Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(); + private DevelopmentService service; + + @Override + public OrchestratorExportResponseRef exportRef(OrchestratorExportRequestRef requestRef) throws ExternalOperationFailedException { + if (null == requestRef){ + LOGGER.error("request ref for exporting is null, it is a fatal error"); + return null; + } + LOGGER.info("Begin to ask to export orchestrator, requestRef is {}", DSSCommonUtils.COMMON_GSON.toJson(requestRef)); + RequestExportOrchestrator exportRequest = new RequestExportOrchestrator(requestRef.getUserName(), + requestRef.getWorkspaceName(), requestRef.getOrcId(), -1L, + requestRef.getProjectName(), requestRef.getDSSLabels(), requestRef.getAddOrcVersionFlag(), + BDPJettyServerHelper.gson().toJson(requestRef.getWorkspace())); + ResponseExportOrchestrator exportResponse = null; + try{ + exportResponse = (ResponseExportOrchestrator) sender.ask(exportRequest); + }catch(final Throwable e){ + DSSExceptionUtils.dealErrorException(60015, "export orchestrator ref failed", e, + ExternalOperationFailedException.class); + } + LOGGER.info("End to ask to export orchestrator, responseRef is {}", DSSCommonUtils.COMMON_GSON.toJson(exportResponse)); + if(exportResponse == null){ + LOGGER.error("exportResponse is null, it means export is failed"); + DSSExceptionUtils.dealErrorException(63323, "exportResponse is null, it means export is failed", ExternalOperationFailedException.class); + } + DefaultOrchestratorExportResponseRef exportResponseRef = new DefaultOrchestratorExportResponseRef(); + exportResponseRef.setBmlVersion(exportResponse.version()); + exportResponseRef.setResourceId(exportResponse.resourceId()); + exportResponseRef.setOrchestratorVersionId(exportResponse.orcVersionId()); + return exportResponseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.service = service; + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkImportOperation.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkImportOperation.java new file mode 100644 index 000000000..c18b0732a --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkImportOperation.java @@ -0,0 +1,71 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.operation; + +import com.webank.wedatasphere.dss.common.protocol.ResponseImportOrchestrator; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestImportOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorImportRequestRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class OrchestratorFrameworkImportOperation implements + RefImportOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorFrameworkImportOperation.class); + private DevelopmentService service; + + @Override + public CommonResponseRef importRef(OrchestratorImportRequestRef requestRef) throws ExternalOperationFailedException { + if (null == requestRef) { + return null; + } + LOGGER.info("Begin to ask to import orchestrator, requestRef is {}", DSSCommonUtils.COMMON_GSON.toJson(requestRef)); + RequestImportOrchestrator importRequest = new RequestImportOrchestrator(requestRef.getUserName(), + requestRef.getWorkspaceName(), requestRef.getProjectName(), + requestRef.getProjectId(), requestRef.getResourceId(), + requestRef.getBmlVersion(), requestRef.getOrcName(), requestRef.getDSSLabels(), + DSSCommonUtils.COMMON_GSON.toJson(requestRef.getWorkspace())); + ResponseImportOrchestrator importResponse = null; + try { + Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(requestRef.getDSSLabels()); + importResponse = (ResponseImportOrchestrator) sender.ask(importRequest); + } catch (final Throwable t) { + DSSExceptionUtils.dealErrorException(60015, "import orchestrator ref failed", t, + ExternalOperationFailedException.class); + } + LOGGER.info("End to ask to import orchestrator, responseRef is {}", DSSCommonUtils.COMMON_GSON.toJson(importResponse)); + CommonResponseRef importResponseRef = new CommonResponseRef(); + if (null == importResponse){ + LOGGER.error("importResponse is null it means failed to import Ref"); + DSSExceptionUtils.dealErrorException(60015, "import ref response is null", ExternalOperationFailedException.class); + } + importResponseRef.setOrcId(importResponse.orcId()); + return importResponseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.service = service; + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkQueryOperation.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkQueryOperation.java new file mode 100644 index 000000000..4dc75bb62 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkQueryOperation.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.operation; + +import com.webank.wedatasphere.dss.appconn.orchestrator.ref.DefaultOrchestratorQueryResponseRef; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestQueryOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseQueryOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorQueryRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorQueryResponseRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class OrchestratorFrameworkQueryOperation implements RefQueryOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorFrameworkQueryOperation.class); + + private final Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(); + private DevelopmentService developmentService; + + @Override + public OrchestratorQueryResponseRef query(OrchestratorQueryRequestRef requestRef) throws ExternalOperationFailedException { + if (null == requestRef) { + LOGGER.error("request of query is null"); + return null; + } + LOGGER.info("Begin to ask to create orchestrator, requestRef is {}", requestRef); + RequestQueryOrchestrator queryRequest = new RequestQueryOrchestrator(requestRef.getOrchestratorIdList()); + ResponseQueryOrchestrator queryResponse = null; + try { + queryResponse = (ResponseQueryOrchestrator) sender.ask(queryRequest); + } catch (Exception e) { + DSSExceptionUtils.dealErrorException(60015, "create orchestrator ref failed", + ExternalOperationFailedException.class); + } + if (queryResponse == null) { + LOGGER.error("query response is null, it is a fatal error"); + return null; + } + LOGGER.info("End to ask to query orchestrator, responseRef is {}", queryResponse); + OrchestratorQueryResponseRef queryResponseRef = new DefaultOrchestratorQueryResponseRef(); + queryResponseRef.setOrchestratorVoList(queryResponse.getOrchestratorVoes()); + return queryResponseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkUpdateOperation.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkUpdateOperation.java new file mode 100644 index 000000000..a69671bd1 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/operation/OrchestratorFrameworkUpdateOperation.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.operation; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestUpdateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorUpdateRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class OrchestratorFrameworkUpdateOperation implements + RefUpdateOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorFrameworkUpdateOperation.class); + + @Override + public CommonResponseRef updateRef(OrchestratorUpdateRef requestRef) throws ExternalOperationFailedException { + if (null == requestRef){ + LOGGER.error("request ref is null, can not deal with null ref"); + return null; + } + LOGGER.info("Begin to ask to update orchestrator, requestRef is {}", requestRef); + RequestUpdateOrchestrator updateRequest = new RequestUpdateOrchestrator(requestRef.getUserName(), + requestRef.getWorkspaceName(), requestRef.getOrchestratorInfo(), requestRef.getDSSLabels()); + ResponseOperateOrchestrator updateResponse = null; + Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(requestRef.getDSSLabels()); + try{ + updateResponse = (ResponseOperateOrchestrator) sender.ask(updateRequest); + }catch(final Exception e){ + DSSExceptionUtils.dealErrorException(60015, "update orchestrator ref failed", e, + ExternalOperationFailedException.class); + } + LOGGER.info("End to ask to update orchestrator, responseRef is {}", updateResponse); + if (updateResponse == null) { + LOGGER.error("to get updateResponse from orchestrator is null"); + return null; + } + CommonResponseRef updateResponseRef = new CommonResponseRef(); + updateResponseRef.setResult(JobStatus.Success.equals(updateResponse.getJobStatus())); + return updateResponseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorCreateResponseRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorCreateResponseRef.java new file mode 100644 index 000000000..2ee515573 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorCreateResponseRef.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + +public class DefaultOrchestratorCreateResponseRef extends CommonResponseRef implements OrchestratorCreateResponseRef { + + + private Long versionId; + + + @Override + public Long getOrchestratorVersionId() { + return versionId; + } + + @Override + public void setOrchestratorVersionId(Long versionId) { + this.versionId = versionId; + } + + + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorDeleteRequestRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorDeleteRequestRef.java new file mode 100644 index 000000000..5d8f1dfe1 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorDeleteRequestRef.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorDeleteRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + +public class DefaultOrchestratorDeleteRequestRef extends CommonRequestRefImpl implements OrchestratorDeleteRequestRef { + + private Long appId; + + @Override + public void setAppId(Long appId) { + this.appId = appId; + } + + @Override + public Long getAppId() { + return this.appId; + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorExportRequestRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorExportRequestRef.java new file mode 100644 index 000000000..937c73480 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorExportRequestRef.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + +public class DefaultOrchestratorExportRequestRef extends CommonRequestRefImpl implements OrchestratorExportRequestRef { + + + private Long appId; + + private Long orchestratorVersionId; + + private boolean addOrcVersion; + + + @Override + public void setAppId(Long appId) { + this.appId = appId; + } + + @Override + public Long getAppId() { + return appId; + } + + @Override + public void setOrchestratorVersionId(Long orchestratorVersionId) { + this.orchestratorVersionId = orchestratorVersionId; + } + + + @Override + public Long getOrchestratorVersionId() { + return orchestratorVersionId; + } + + + @Override + public boolean getAddOrcVersionFlag() { + return addOrcVersion; + } + + @Override + public void setAddOrcVersionFlag(boolean addOrcVersion) { + this.addOrcVersion = addOrcVersion; + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorExportResponseRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorExportResponseRef.java new file mode 100644 index 000000000..3464e0af6 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorExportResponseRef.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + + +public class DefaultOrchestratorExportResponseRef extends CommonResponseRef implements OrchestratorExportResponseRef { + + private String resourceId; + + private String bmlVersion; + + private Long orchestratorVersionId; + @Override + public String getResourceId() { + return resourceId; + } + + @Override + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + @Override + public String getBmlVersion() { + return bmlVersion; + } + + @Override + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + @Override + public Long getOrchestratorVersionId() { + return orchestratorVersionId; + } + + @Override + public void setOrchestratorVersionId(Long orchestratorVersionId) { + this.orchestratorVersionId = orchestratorVersionId; + } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorImportRequestRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorImportRequestRef.java new file mode 100644 index 000000000..ff65b14c1 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorImportRequestRef.java @@ -0,0 +1,70 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorImportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class DefaultOrchestratorImportRequestRef extends CommonRequestRefImpl implements OrchestratorImportRequestRef { + private String resourceId; + private String bmlVersion; + private IOEnv sourceEnv; + private String orcVersion; + + @Override + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + @Override + public String getResourceId() { + return resourceId; + } + + @Override + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + @Override + public String getBmlVersion() { + return bmlVersion; + } + @Override + public void setSourceEnv(IOEnv sourceEnv) { + this.sourceEnv = sourceEnv; + } + + @Override + public IOEnv getSourceEnv() { + return sourceEnv; + } + + @Override + public void setOrcVersion(String orcVersion) { + this.orcVersion = orcVersion; + } + + @Override + public String getOrcVersion() { + return orcVersion; + } + + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorQueryRequestRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorQueryRequestRef.java new file mode 100644 index 000000000..164f2f8d6 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorQueryRequestRef.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorQueryRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + +import java.util.List; + + +public class DefaultOrchestratorQueryRequestRef extends CommonRequestRefImpl implements OrchestratorQueryRequestRef { + + private List orcIdList; + + @Override + public List getOrchestratorIdList() { + return orcIdList; + } + + @Override + public void setOrchestratorIdList(List orchestratorIdList) { + this.orcIdList = orchestratorIdList; + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorQueryResponseRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorQueryResponseRef.java new file mode 100644 index 000000000..bd7cead0b --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorQueryResponseRef.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorQueryResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + +import java.util.List; + +public class DefaultOrchestratorQueryResponseRef extends CommonResponseRef implements OrchestratorQueryResponseRef { + private List orchestratorVos; + + @Override + public List getOrchestratorVos() { + return orchestratorVos; + } + + @Override + public void setOrchestratorVoList(List orchestratorVoList) { + this.orchestratorVos = orchestratorVoList; + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorUpdateRequestRef.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorUpdateRequestRef.java new file mode 100644 index 000000000..b2f903fd8 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/ref/DefaultOrchestratorUpdateRequestRef.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorUpdateRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + +public class DefaultOrchestratorUpdateRequestRef extends CommonRequestRefImpl implements OrchestratorUpdateRef { + + private String description; + private String uses; + + private DSSOrchestratorInfo dssOrchestratorInfo; + + @Override + public void setDescription(String description) { + this.description = description; + } + + @Override + public String getDescription() { + return description; + } + + @Override + public void setUses(String uses) { + this.uses = uses; + } + + @Override + public String getUses() { + return uses; + } + + @Override + public DSSOrchestratorInfo getOrchestratorInfo() { + return null; + } + + @Override + public void setOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo = dssOrchestratorInfo; } +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorCRUDService.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorCRUDService.java new file mode 100644 index 000000000..920a2706b --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorCRUDService.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.service; + +import com.webank.wedatasphere.dss.appconn.orchestrator.operation.OrchestratorFrameworkCreationOperation; +import com.webank.wedatasphere.dss.appconn.orchestrator.operation.OrchestratorFrameworkDeleteOperation; +import com.webank.wedatasphere.dss.appconn.orchestrator.operation.OrchestratorFrameworkUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCopyOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefCRUDService; + +public class OrchestratorCRUDService extends AbstractRefCRUDService { + + @Override + protected RefCreationOperation createRefCreationOperation() { + return new OrchestratorFrameworkCreationOperation(); + } + + @Override + protected RefCopyOperation createRefCopyOperation() { + return null; + } + + @Override + protected RefUpdateOperation createRefUpdateOperation() { + return new OrchestratorFrameworkUpdateOperation(); + } + + @Override + protected RefDeletionOperation createRefDeletionOperation() { + return new OrchestratorFrameworkDeleteOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorExportProcessService.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorExportProcessService.java new file mode 100644 index 000000000..2b0fb967e --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorExportProcessService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.service; + +import com.webank.wedatasphere.dss.appconn.orchestrator.operation.OrchestratorFrameworkExportOperation; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExportService; + +public class OrchestratorExportProcessService extends AbstractRefExportService { + + @Override + @SuppressWarnings("unchecked") + public RefExportOperation createRefExportOperation() { + return new OrchestratorFrameworkExportOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorImportProcessService.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorImportProcessService.java new file mode 100644 index 000000000..88b364c2e --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorImportProcessService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.service; + +import com.webank.wedatasphere.dss.appconn.orchestrator.operation.OrchestratorFrameworkImportOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefImportService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; + +public class OrchestratorImportProcessService extends AbstractRefImportService { + + @Override + protected RefImportOperation createRefImportOperation() { + return new OrchestratorFrameworkImportOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorQueryService.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorQueryService.java new file mode 100644 index 000000000..11649efd3 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/service/OrchestratorQueryService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.service; + +import com.webank.wedatasphere.dss.appconn.orchestrator.operation.OrchestratorFrameworkQueryOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefQueryService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; + +public class OrchestratorQueryService extends AbstractRefQueryService { + + @Override + public RefQueryOperation createRefQueryOperation() { + OrchestratorFrameworkQueryOperation orchestratorFrameworkQueryOperation = new OrchestratorFrameworkQueryOperation(); + orchestratorFrameworkQueryOperation.setDevelopmentService(this); + return orchestratorFrameworkQueryOperation ; + } + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/standard/OrchestratorFrameworkStandard.java b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/standard/OrchestratorFrameworkStandard.java new file mode 100644 index 000000000..dfd6072bc --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/orchestrator/standard/OrchestratorFrameworkStandard.java @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.orchestrator.standard; + +import com.webank.wedatasphere.dss.appconn.orchestrator.service.OrchestratorCRUDService; +import com.webank.wedatasphere.dss.appconn.orchestrator.service.OrchestratorExportProcessService; +import com.webank.wedatasphere.dss.appconn.orchestrator.service.OrchestratorImportProcessService; +import com.webank.wedatasphere.dss.appconn.orchestrator.service.OrchestratorQueryService; +import com.webank.wedatasphere.dss.standard.app.development.service.*; +import com.webank.wedatasphere.dss.standard.app.development.standard.AbstractDevelopmentIntegrationStandard; + +public class OrchestratorFrameworkStandard extends AbstractDevelopmentIntegrationStandard { + + private volatile static OrchestratorFrameworkStandard instance; + + public static OrchestratorFrameworkStandard getInstance(){ + if (instance == null){ + synchronized (OrchestratorFrameworkStandard.class){ + if (instance == null){ + instance = new OrchestratorFrameworkStandard(); + } + } + } + return instance; + } + + @Override + protected RefCRUDService createRefCRUDService() { + return new OrchestratorCRUDService(); + } + + @Override + protected RefExecutionService createRefExecutionService() { + return null; + } + + @Override + protected RefExportService createRefExportService() { + return new OrchestratorExportProcessService(); + } + + @Override + protected RefImportService createRefImportService() { + return new OrchestratorImportProcessService(); + } + + @Override + protected RefQueryService createRefQueryService() { + return new OrchestratorQueryService(); + } + + + @Override + public String getStandardName() { + return "OrchestratorFrameworkStandard"; + } + + +} diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/appconn.properties b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/appconn.properties new file mode 100644 index 000000000..19365e2b5 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/appconn.properties @@ -0,0 +1,20 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + + + + + diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/log4j.properties b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/log4j.properties new file mode 100644 index 000000000..ee8619595 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO diff --git a/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/log4j2.xml b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/log4j2.xml new file mode 100644 index 000000000..8c40a73e8 --- /dev/null +++ b/dss-appconn/appconns/dss-orchestrator-framework-appconn/src/main/resources/log4j2.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-appconn/appconns/dss-schedulis-appconn/pom.xml b/dss-appconn/appconns/dss-schedulis-appconn/pom.xml new file mode 100644 index 000000000..926ad110c --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/pom.xml @@ -0,0 +1,136 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-schedulis-appconn + + + + + + com.webank.wedatasphere.dss + dss-scheduler-appconn + ${dss.version} + + + + + com.webank.wedatasphere.dss + dss-origin-sso-integration-standard + ${dss.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + + org.apache.httpcomponents + httpclient + 4.5.13 + + + + + com.google.code.gson + gson + ${gson.version} + provided + + + com.webank.wedatasphere.dss + dss-contextservice + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-schedulis-appconn/src/main/assembly/distribution.xml new file mode 100644 index 000000000..36cd42577 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/assembly/distribution.xml @@ -0,0 +1,76 @@ + + + + dss-schedulis-appconn + + dir + + true + schedulis + + + + + + lib + true + true + false + true + true + + + + + + ${basedir}/src/main/resources + + appconn.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + log4j.properties + log4j2.xml + + 0777 + conf + unix + + + + ${basedir}/src/main/resources + + init.sql + + 0777 + db + + + + + + + diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleAction.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleAction.java new file mode 100644 index 000000000..17611d6be --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleAction.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.Action; + +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction; + +public class FlowScheduleAction extends POSTAction { + + private String url; + @Override + public String getRequestPayload() { + return ""; + } + + public void setURL(String url) { + this.url = url; + } + + @Override + public String getURL() { + return url; + } +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleGetAction.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleGetAction.java new file mode 100644 index 000000000..e316579cf --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleGetAction.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.Action; + +import com.webank.wedatasphere.linkis.httpclient.request.GetAction; +import com.webank.wedatasphere.linkis.httpclient.request.UserAction; + +public class FlowScheduleGetAction extends GetAction implements UserAction { + + String url; + String user; + + @Override + public String getURL() { + return url; + } + + public void setURL(String url) { + this.url = url; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + @Override + public String getUser() { + return user; + } + + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleUploadAction.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleUploadAction.java new file mode 100644 index 000000000..844a7494f --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/Action/FlowScheduleUploadAction.java @@ -0,0 +1,105 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.Action; + +import com.webank.wedatasphere.dss.appconn.schedulis.sso.UserInfo; +import com.webank.wedatasphere.linkis.httpclient.request.BinaryBody; +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction; +import com.webank.wedatasphere.linkis.httpclient.request.UploadAction; +import scala.Option; + + +import java.io.InputStream; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +public class FlowScheduleUploadAction extends POSTAction implements UploadAction, UserInfo { + private Map inputStreams; + private List binaryBodies; + private Map streamNames=new HashMap<>(); + + private String url; + private String user; + private ArrayList filePaths; + + public FlowScheduleUploadAction(List binaryBodies){ + this.filePaths=null; + this.binaryBodies = binaryBodies; + } + + @Override + public String getURL() { + return this.url; + } + + public void setURl(String url){ + this.url = url; + } + + @Override + public Map files() { + Map map = new HashMap<>(); + + if (null == filePaths || filePaths.size() == 0) { + return map; + } + else { + filePaths.stream().forEach( + filePath -> map.put("file", filePath)); + } + + return map; + } + + @Override + public Map inputStreams() { + return inputStreams; + } + + @Override + public Map inputStreamNames() { + return streamNames; + } + + @Override + public Option user() { + return null; + } + + + @Override + public String getRequestPayload() { + return null; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + @Override + public String getUser() { + return user; + } + + @Override + public List binaryBodies() { + return binaryBodies; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/SchedulisAppConn.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/SchedulisAppConn.java new file mode 100644 index 000000000..33f1a7e24 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/SchedulisAppConn.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis; + +import com.webank.wedatasphere.dss.appconn.scheduler.AbstractSchedulerAppConn; +import com.webank.wedatasphere.dss.appconn.schedulis.standard.SchedulisStructureStandard; +import com.webank.wedatasphere.dss.standard.app.structure.StructureIntegrationStandard; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class SchedulisAppConn extends AbstractSchedulerAppConn { + + public static final String SCHEDULIS_APPCONN_NAME = "Schedulis"; + + @Override + public StructureIntegrationStandard getOrCreateStructureStandard() { + return SchedulisStructureStandard.getInstance(); + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conf/AzkabanConf.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conf/AzkabanConf.java new file mode 100644 index 000000000..7bd7e58de --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conf/AzkabanConf.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conf; + + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +public class AzkabanConf { + public static final CommonVars DEFAULT_STORE_PATH = CommonVars.apply("wds.dss.appconn.scheduler.project.store.dir", "/appcom/tmp/wds/dss"); + public static final CommonVars AZKABAN_LOGIN_PWD = CommonVars.apply("wds.dss.appconn.scheduler.azkaban.login.passwd", "userpwd"); + public static final CommonVars LINKIS_VERSION = CommonVars.apply("wds.dss.appconn.scheduler.linkis.version", "1.0.0"); + public static final CommonVars JOB_LABEL = CommonVars.apply("wds.dss.appconn.scheduler.job.label", "prod"); + + public static final CommonVars AZKABAN_RSA = + CommonVars.apply("wds.dss.appconn.scheduler.azkaban.rsa", + "iVGljygYsZvrNFCYqOpocm4Kpp7rJudeKomnM9FImWI4yNjdtmOt43Q1Brb7MFgzjRLIdJWJ0Ui760pDZpXPHOz81ctsj553E5cdKu7cg+h5C2AkBEZ6AvEQ+oet7ukwg+7ASSBuQufLkAxHKDqCjq/XxsC/MH11pkuHKaJpSTY="); + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/constant/AzkabanConstant.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/constant/AzkabanConstant.java new file mode 100644 index 000000000..92b73191d --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/constant/AzkabanConstant.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.constant; + +public class AzkabanConstant { + public final static String LINKIS_FLOW_VARIABLE_KEY = "flow.variable."; + public final static String AZKABAN_JOB_SUFFIX = ".job"; + public final static String AZKABAN_PROPERTIES_SUFFIX = ".properties"; + public final static String LINKIS_JOB_RESOURCES_KEY = "resources="; + public final static String ZAKABAN_DEPENDENCIES_KEY = "dependencies"; + public final static String JOB_TYPE = "type"; + public final static String JOB_LABELS = "labels"; + public final static String LINKIS_TYPE = "linkistype"; + public final static String JOB_COMMAND = "command"; + public final static String FLOW_CONTEXT_ID = "wds.linkis.flow.contextID="; + public final static String LINKIS_VERSION = "linkis.version"; + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/AzkabanWorkflowToRelSynchronizer.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/AzkabanWorkflowToRelSynchronizer.java new file mode 100644 index 000000000..90e06d22e --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/AzkabanWorkflowToRelSynchronizer.java @@ -0,0 +1,110 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conversion; + +import com.webank.wedatasphere.dss.appconn.schedulis.Action.FlowScheduleUploadAction; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanConvertedRel; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.SSORequestWTSS; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.common.utils.ZipHelper; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.operation.WorkflowToRelSynchronizer; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.httpclient.request.BinaryBody; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import java.io.File; +import java.io.FileInputStream; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang.exception.ExceptionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class AzkabanWorkflowToRelSynchronizer implements WorkflowToRelSynchronizer { + + public static final Logger LOGGER = LoggerFactory.getLogger(AzkabanWorkflowToRelSynchronizer.class); + + private String projectUrl; + private SSORequestService ssoRequestService; + + @Override + public void setAppInstance(AppInstance appInstance) { + this.projectUrl = appInstance.getBaseUrl().endsWith("/") ? appInstance.getBaseUrl() + "manager": + appInstance.getBaseUrl() + "/manager"; + } + + @Override + public void syncToRel(ConvertedRel convertedRel) { + String tmpSavePath; + AzkabanConvertedRel azkabanConvertedRel = (AzkabanConvertedRel) convertedRel; + try { + String projectPath = azkabanConvertedRel.getStorePath(); + tmpSavePath = ZipHelper.zip(projectPath); + //upload zip to Azkaban + uploadProject(azkabanConvertedRel.getDSSToRelConversionRequestRef().getWorkspace(), tmpSavePath, + azkabanConvertedRel.getDSSToRelConversionRequestRef().getDSSProject().getName(), azkabanConvertedRel.getDSSToRelConversionRequestRef().getUserName()); + } catch (Exception e) { + throw new DSSRuntimeException(90012, ExceptionUtils.getRootCauseMessage(e), e); + } + } + + @Override + public void setSSORequestService(SSORequestService ssoRequestService) { + this.ssoRequestService = ssoRequestService; + } + + private void uploadProject(Workspace workspace, String tmpSavePath, String projectName, String releaseUser) throws Exception { + + File file = new File(tmpSavePath); + InputStream inputStream = new FileInputStream(file); + try { + BinaryBody binaryBody =BinaryBody.apply("file",inputStream,file.getName(),"application/zip"); + List binaryBodyList =new ArrayList<>(); + binaryBodyList.add(binaryBody); + FlowScheduleUploadAction uploadAction = new FlowScheduleUploadAction(binaryBodyList); + uploadAction.getFormParams().put("ajax", "upload"); + uploadAction.getFormParams().put("project", projectName); + + uploadAction.getParameters().put("project", projectName); + uploadAction.getParameters().put("ajax", "upload"); + uploadAction.setURl(projectUrl); + + + HttpResult response = SSORequestWTSS.requestWTSSWithSSOUpload(projectUrl,uploadAction,this.ssoRequestService,workspace); + + if (response.getStatusCode() == 200 || response.getStatusCode()==0) { + LOGGER.info("upload project:{} success!", projectName); + }else{ + LOGGER.error("调用azkaban上传接口的返回不为200, status code 是 {}", response.getStatusCode()); + throw new ErrorException(90013, "release project failed, " + response.getResponseBody()); + } + + } catch (Exception e) { + LOGGER.error("upload failed,reason:", e); + throw new ErrorException(90014, e.getMessage()); + } finally { + IOUtils.closeQuietly(inputStream); + } + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/NodeConverter.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/NodeConverter.java new file mode 100644 index 000000000..605c61154 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/NodeConverter.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conversion; + +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; + +public interface NodeConverter { + + String conversion(WorkflowNode workflowNode); + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/ProjectInfoWorkflowToRelConverter.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/ProjectInfoWorkflowToRelConverter.java new file mode 100644 index 000000000..681a695d6 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/ProjectInfoWorkflowToRelConverter.java @@ -0,0 +1,109 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conversion; + +import com.webank.wedatasphere.dss.appconn.schedulis.conf.AzkabanConf; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanConvertedRel; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanWorkflow; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.AzkabanUtilsScala; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRel; +import com.webank.wedatasphere.dss.workflow.conversion.operation.WorkflowToRelConverter; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import java.io.File; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import org.apache.commons.io.FileUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class ProjectInfoWorkflowToRelConverter implements WorkflowToRelConverter { + + private static final Logger LOGGER = LoggerFactory.getLogger(ProjectInfoWorkflowToRelConverter.class); + + @Override + public ConvertedRel convertToRel(PreConversionRel rel) { + List repeatNode = AzkabanUtilsScala.getRepeatNodeName(getAllNodeName(rel.getWorkflows())); + if (!repeatNode.isEmpty()) { + throw new DSSRuntimeException(80001, "重复的节点名称：" + repeatNode.toString()); + } + AzkabanConvertedRel azkabanConvertedRel = new AzkabanConvertedRel(rel); + //1. Assign a value to the storepath of azkabanschedulerproject. + assignStorePath(azkabanConvertedRel); + //2. The storepath of his rootflow is also assigned a value. + List workflows = rel.getWorkflows(); + workflows.forEach(flow -> setRootFlowStorePath(azkabanConvertedRel.getStorePath(), flow)); + //3. Delete zip packages and folders that may not have been processed. + removeProjectStoreDirAndZip(azkabanConvertedRel); + return azkabanConvertedRel; + } + + private void setRootFlowStorePath(String projectStorePath, Workflow workflow){ + AzkabanWorkflow azkabanWorkflow = (AzkabanWorkflow) workflow; + azkabanWorkflow.setStorePath(projectStorePath + File.separator + workflow.getName()); + } + + private void assignStorePath(AzkabanConvertedRel rel) { + SimpleDateFormat dateFormat = new SimpleDateFormat(AzkabanConvertedRel.DATE_FORMAT); + Date date = new Date(); + String dateStr = dateFormat.format(date); + String userName = rel.getDSSToRelConversionRequestRef().getDSSProject().getUsername(); + String name = rel.getDSSToRelConversionRequestRef().getDSSProject().getName(); + String storePath = AzkabanConf.DEFAULT_STORE_PATH.getValue() + File.separator + userName + + File.separator + dateStr + File.separator +name; + rel.setStorePath(storePath); + } + + private void removeProjectStoreDirAndZip(AzkabanConvertedRel rel) { + String storePath = rel.getStorePath(); + File projectDir = new File(storePath); + try { + if (projectDir.exists()) { + LOGGER.info("exist project dir{} before publish ,now remove it", storePath); + FileUtils.deleteDirectory(projectDir); + } + String projectZip = projectDir.getParent() + File.separator + + rel.getDSSToRelConversionRequestRef().getDSSProject().getName() + ".zip"; + File zipFile = new File(projectZip); + if (zipFile.exists()) { + LOGGER.info("exist project zip{} before publish ,now remove it", projectZip); + zipFile.delete(); + } + } catch (Exception e) { + LOGGER.error("delete project dir or zip failed,reaseon:", e); + throw new DSSRuntimeException(90020, e.getMessage()); + } + } + + /** + * Get the names of all nodes directly from allflows without recursion. + */ + private List getAllNodeName(List workflows) { + List nodeNames = new ArrayList<>(); + workflows.forEach(flow -> flow.getWorkflowNodes().forEach(node -> nodeNames.add(node.getName()))); + return nodeNames; + } + + @Override + public int getOrder() { + return 5; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/WorkflowToAzkbanNodeRelConverter.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/WorkflowToAzkbanNodeRelConverter.java new file mode 100644 index 000000000..dcb02ad85 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/WorkflowToAzkbanNodeRelConverter.java @@ -0,0 +1,111 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conversion; + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.appconn.schedulis.constant.AzkabanConstant; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanWorkflow; +import com.webank.wedatasphere.dss.appconn.schedulis.linkisjob.LinkisJobConverter; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRel; +import com.webank.wedatasphere.dss.workflow.conversion.operation.WorkflowToRelConverter; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import java.io.File; +import java.io.FileOutputStream; +import java.util.List; +import org.apache.commons.io.FileUtils; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class WorkflowToAzkbanNodeRelConverter implements WorkflowToRelConverter { + + public static final Logger LOGGER = LoggerFactory.getLogger(WorkflowToAzkbanNodeRelConverter.class); + + private NodeConverter nodeConverter; + + public WorkflowToAzkbanNodeRelConverter() { + nodeConverter = ClassUtils.getInstanceOrDefault(NodeConverter.class, new LinkisJobConverter()); + } + + @Override + public ConvertedRel convertToRel(PreConversionRel rel) { + rel.getWorkflows().forEach(this::convertNode); + return (ConvertedRel) rel; + } + + private void convertNode(Workflow workflow) { + workflow.getWorkflowNodes().forEach(DSSExceptionUtils.handling(workflowNode -> { + String nodeStorePath = getNodeStorePath(((AzkabanWorkflow)workflow).getStorePath(), workflowNode); + writeNodeToJobLocal(workflowNode, nodeStorePath); + writeNodeResourcesToLocal(workflowNode, nodeStorePath); + })); + if(workflow.getChildren() != null) { + workflow.getChildren().forEach(flow -> convertNode((Workflow) flow)); + } + } + + private String getNodeStorePath(String flowStorePath, WorkflowNode schedulerNode) { + return flowStorePath + File.separator + "jobs" + File.separator + schedulerNode.getName(); + } + + private void writeNodeToJobLocal(WorkflowNode workflowNode, String storePath) throws DSSErrorException { + FileOutputStream os = null; + try { + File jobDirFile = new File(storePath); + FileUtils.forceMkdir(jobDirFile); + File jobFile = new File(storePath,workflowNode.getName() + AzkabanConstant.AZKABAN_JOB_SUFFIX); + jobFile.createNewFile(); + String nodeString = nodeConverter.conversion(workflowNode); + os = FileUtils.openOutputStream(jobFile,true); + os.write(nodeString.getBytes()); + }catch (Exception e){ + LOGGER.error("write AppConnNode to jobLocal failed,reason:",e); + throw new DSSErrorException(90017,e.getMessage()); + } finally { + IOUtils.closeQuietly(os); + } + } + + private void writeNodeResourcesToLocal(WorkflowNode workflowNode, String storePath) throws DSSErrorException { + List nodeResources = workflowNode.getDSSNode().getResources(); + if(nodeResources == null || nodeResources.isEmpty()) {return;} + FileOutputStream os = null; + try { + File jobFile = new File(storePath,workflowNode.getName() + AzkabanConstant.AZKABAN_JOB_SUFFIX); + String nodeResourceString = AzkabanConstant.LINKIS_JOB_RESOURCES_KEY + new Gson().toJson(nodeResources); + os = FileUtils.openOutputStream(jobFile,true); + os.write(nodeResourceString.getBytes()); + }catch (Exception e){ + LOGGER.error("write nodeResources to local failed,reason:",e); + throw new DSSErrorException(90018,e.getMessage()); + }finally { + IOUtils.closeQuietly(os); + } + } + + @Override + public int getOrder() { + return 100; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/WorkflowToAzkbanRelConverter.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/WorkflowToAzkbanRelConverter.java new file mode 100644 index 000000000..a413d9d93 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/conversion/WorkflowToAzkbanRelConverter.java @@ -0,0 +1,142 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conversion; + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.appconn.schedulis.constant.AzkabanConstant; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanConvertedRel; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanWorkflow; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.workflow.common.entity.Flow; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRel; +import com.webank.wedatasphere.dss.workflow.conversion.operation.WorkflowToRelConverter; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import java.io.File; +import java.io.FileOutputStream; +import java.util.List; +import java.util.Map; +import org.apache.commons.io.FileUtils; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class WorkflowToAzkbanRelConverter implements WorkflowToRelConverter { + + public static final Logger LOGGER = LoggerFactory.getLogger(WorkflowToAzkbanRelConverter.class); + + @Override + public ConvertedRel convertToRel(PreConversionRel rel) { + AzkabanConvertedRel azkabanConvertedRel = (AzkabanConvertedRel) rel; + azkabanConvertedRel.getWorkflows().forEach(DSSExceptionUtils.handling(workflow -> { + //1. Set sub flow and node storage paths + String flowStorePath = ((AzkabanWorkflow) workflow).getStorePath(); + if (workflow.getChildren() != null) { + workflow.getChildren().forEach(flow -> setFlowStorePath(flowStorePath, flow)); + } + // 2. Processing resources, generating files, and so on. + writeWorkflowFiles(workflow, azkabanConvertedRel.getStorePath()); + })); + return azkabanConvertedRel; + } + + private void writeWorkflowFiles(Flow workflow, String projectStorePath) throws DSSErrorException { + AzkabanWorkflow flow = (AzkabanWorkflow) workflow; + writeFlowResourcesToLocal(flow, projectStorePath); + writeFlowPropertiesToLocal(flow); + if (workflow.getChildren() != null) { + workflow.getChildren().forEach(DSSExceptionUtils.handling(f -> writeWorkflowFiles(f, projectStorePath))); + } + } + + private void setFlowStorePath(String flowStorePath, Flow workflow) { + AzkabanWorkflow azkabanWorkflow = (AzkabanWorkflow) workflow; + azkabanWorkflow.setStorePath(flowStorePath + File.separator + "subFlows" + File.separator + azkabanWorkflow.getName()); + if (workflow.getChildren() != null) { + workflow.getChildren().forEach(flow -> setFlowStorePath(azkabanWorkflow.getStorePath(), flow)); + } + } + + private void writeFlowResourcesToLocal(AzkabanWorkflow flow, String projectStorePath) throws DSSErrorException { + List flowResources = flow.getFlowResources(); + FileOutputStream os = null; + try { + String storePath = flow.getStorePath(); + File flowDir = new File(storePath); + FileUtils.forceMkdir(flowDir); + if (flowResources == null || flowResources.isEmpty()) { + return; + } + String flowResourceStringPrefix = getFlowResourceStringPrefix(projectStorePath, storePath); + String flowResourceString = flowResourceStringPrefix + new Gson().toJson(flowResources) + "\n"; + File projectResourcesFile = new File(projectStorePath, "project.properties"); + os = FileUtils.openOutputStream(projectResourcesFile, true); + os.write(flowResourceString.getBytes()); + } catch (Exception e) { + LOGGER.error("write FlowResources to local failed,reason:", e); + throw new DSSErrorException(90006, e.getMessage()); + } finally { + IOUtils.closeQuietly(os); + } + } + + private String getFlowResourceStringPrefix(String projectStorePath, String storePath) { + String substring = storePath.substring(projectStorePath.length() + 1); + String prefix = substring.replaceAll("\\" + File.separator + "subFlows" + "\\" + File.separator, "."); + return "flow." + prefix + "_.resources="; + } + + private void writeFlowPropertiesToLocal(AzkabanWorkflow flow) throws DSSErrorException { + List> flowProperties = flow.getFlowProperties(); + if (flowProperties == null || flowProperties.isEmpty()) { + return; + } + FileOutputStream os = null; + try { + String storePath = flow.getStorePath(); + File flowPrpsFile = new File(storePath, flow.getName() + AzkabanConstant.AZKABAN_PROPERTIES_SUFFIX); + flowPrpsFile.createNewFile(); + os = FileUtils.openOutputStream(flowPrpsFile, true); + StringBuilder stringBuilder = new StringBuilder(); + flowProperties.forEach(p -> p.forEach((k, v) -> { + stringBuilder.append(AzkabanConstant.LINKIS_FLOW_VARIABLE_KEY + k + "=" + v + "\n"); + })); + // update by peaceWong add contextID to Flow properties + String contextID = flow.getContextID(); + if (StringUtils.isNotBlank(contextID)) { + contextID = contextID.replace("\\", "/"); + LOGGER.info("after replace contextID is {}", contextID); + stringBuilder.append(AzkabanConstant.FLOW_CONTEXT_ID + contextID + "\n"); + } + // update end + os.write(stringBuilder.toString().getBytes()); + } catch (Exception e) { + LOGGER.error("write flowProperties to local faailed,reason:", e); + throw new DSSErrorException(90007, e.getMessage()); + } finally { + IOUtils.closeQuietly(os); + } + } + + @Override + public int getOrder() { + return 10; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanConvertedRel.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanConvertedRel.java new file mode 100644 index 000000000..09c3afc9a --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanConvertedRel.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.entity; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.DSSToRelConversionRequestRef; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.ProjectToRelConversionRequestRef; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRelImpl; + +public class AzkabanConvertedRel extends PreConversionRelImpl implements ConvertedRel { + + private String storePath; + public static final String DATE_FORMAT = "yyyyMMddHHmmss"; + + public AzkabanConvertedRel(PreConversionRel rel) { + setWorkflows(rel.getWorkflows()); + setDSSToRelConversionRequestRef(rel.getDSSToRelConversionRequestRef()); + } + + public AzkabanConvertedRel(AzkabanConvertedRel rel) { + this((PreConversionRel) rel); + storePath = rel.getStorePath(); + } + + @Override + public ProjectToRelConversionRequestRef getDSSToRelConversionRequestRef() { + return (ProjectToRelConversionRequestRef) super.getDSSToRelConversionRequestRef(); + } + + public String getStorePath() { + return storePath; + } + + public void setStorePath(String storePath) { + this.storePath = storePath; + } + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanUserEntity.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanUserEntity.java new file mode 100644 index 000000000..d4e839fd6 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanUserEntity.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.entity; + +public class AzkabanUserEntity { + private String id; + private String text; + private String username; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getText() { + return text; + } + + public void setText(String text) { + this.text = text; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanWorkflow.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanWorkflow.java new file mode 100644 index 000000000..1387db767 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/entity/AzkabanWorkflow.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.entity; + +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowWithContextImpl; + +public class AzkabanWorkflow extends WorkflowWithContextImpl { + + private String storePath; + + public String getStorePath() { + return storePath; + } + + public void setStorePath(String storePath) { + this.storePath = storePath; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/AzkabanSubFlowJobTuning.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/AzkabanSubFlowJobTuning.java new file mode 100644 index 000000000..b083be21a --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/AzkabanSubFlowJobTuning.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.linkisjob; + +public class AzkabanSubFlowJobTuning implements LinkisJobTuning { + + @Override + public LinkisJob tuningJob(LinkisJob job) { + job.setType("flow"); + job.setLinkistype(null); + job.getConf().put("flow.name",job.getName() + "_"); + return job; + } + + @Override + public boolean ifJobCantuning(String nodeType) { + return "workflow.subflow".equals(nodeType); + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJob.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJob.java new file mode 100644 index 000000000..5c68f9676 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJob.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.linkisjob; + +import java.util.Map; + +public class LinkisJob { + private String name; + private String type; + private String linkistype; + private String proxyUser; + private String dependencies; + private Map conf; + private String command; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public String getLinkistype() { + return linkistype; + } + + public void setLinkistype(String linkistype) { + this.linkistype = linkistype; + } + + public String getProxyUser() { + return proxyUser; + } + + public void setProxyUser(String proxyUser) { + this.proxyUser = proxyUser; + } + + public String getDependencies() { + return dependencies; + } + + public void setDependencies(String dependencies) { + this.dependencies = dependencies; + } + + public Map getConf() { + return conf; + } + + public void setConf(Map conf) { + this.conf = conf; + } + + public String getCommand() { + return command; + } + + public void setCommand(String command) { + this.command = command; + } + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJobConverter.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJobConverter.java new file mode 100644 index 000000000..69b2cbdde --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJobConverter.java @@ -0,0 +1,127 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.linkisjob; + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.appconn.schedulis.conf.AzkabanConf; +import com.webank.wedatasphere.dss.appconn.schedulis.constant.AzkabanConstant; +import com.webank.wedatasphere.dss.appconn.schedulis.conversion.NodeConverter; +import com.webank.wedatasphere.dss.workflow.core.constant.WorkflowConstant; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.apache.commons.lang.StringUtils; + +public class LinkisJobConverter implements NodeConverter { + + public LinkisJobConverter(){ + LinkisJobTuning[] linkisJobTunings = {new AzkabanSubFlowJobTuning()}; + this.linkisJobTunings = linkisJobTunings; + } + private LinkisJobTuning[] linkisJobTunings; + + @Override + public String conversion(WorkflowNode workflowNode){ + return baseConversion(workflowNode); + } + + private String baseConversion(WorkflowNode workflowNode){ + LinkisJob job = new LinkisJob(); + job.setConf(new HashMap()); + job.setName(workflowNode.getName()); + convertHead(workflowNode,job); + convertDependencies(workflowNode,job); + convertProxyUser(workflowNode,job); + convertConfiguration(workflowNode,job); + convertJobCommand(workflowNode,job); + Arrays.stream(linkisJobTunings).forEach(t ->{ + if(t.ifJobCantuning(workflowNode.getNodeType())) { + t.tuningJob(job); + } + }); + return convertJobToString(job); + } + + private String convertJobToString(LinkisJob job){ + HashMap map = new HashMap<>(); + map.put(AzkabanConstant.LINKIS_VERSION, AzkabanConf.LINKIS_VERSION.getValue()); + map.put(AzkabanConstant.JOB_TYPE,job.getType()); + map.put(AzkabanConstant.LINKIS_TYPE,job.getLinkistype()); + map.put(AzkabanConstant.ZAKABAN_DEPENDENCIES_KEY,job.getDependencies()); + map.put(WorkflowConstant.PROXY_USER,job.getProxyUser()); + map.put(AzkabanConstant.JOB_COMMAND,job.getCommand()); + Map labels = new HashMap(); + labels.put("route",AzkabanConf.JOB_LABEL.getValue()); + map.put(AzkabanConstant.JOB_LABELS, new Gson().toJson(labels)); + map.putAll(job.getConf()); + StringBuilder stringBuilder = new StringBuilder(); + map.forEach((k,v)->{ + if(v != null) { + stringBuilder.append(k).append("=").append(v).append("\n"); + } + }); + return stringBuilder.toString(); + } + + private void convertHead(WorkflowNode workflowNode, LinkisJob job){ + job.setType("linkis"); + job.setLinkistype(workflowNode.getNodeType()); + } + + private void convertDependencies(WorkflowNode workflowNode, LinkisJob job){ + List dependencys = workflowNode.getDSSNode().getDependencys(); + if(dependencys != null && !dependencys.isEmpty()) { + StringBuilder dependencies = new StringBuilder(); + dependencys.forEach(d ->dependencies.append(d + ",")); + job.setDependencies(dependencies.substring(0,dependencies.length()-1)); + } + } + + private void convertProxyUser(WorkflowNode workflowNode, LinkisJob job){ + String userProxy = workflowNode.getDSSNode().getUserProxy(); + if(!StringUtils.isEmpty(userProxy)) { + job.setProxyUser(userProxy); + } + } + + private void convertConfiguration(WorkflowNode workflowNode, LinkisJob job){ + Map params = workflowNode.getDSSNode().getParams(); + if (params != null && !params.isEmpty()) { + Object configuration = params.get("configuration"); + String confprefix = "node.conf."; + ((Map>)configuration).forEach((k,v)-> + { + if(null!=v) { + v.forEach((k2, v2) -> { + if(null !=v2) {job.getConf().put(confprefix + k + "." + k2, v2.toString());} + }); + } + }); + } + + } + + private void convertJobCommand(WorkflowNode workflowNode, LinkisJob job){ + Map jobContent = workflowNode.getDSSNode().getJobContent(); + if(jobContent != null) { + jobContent.remove("jobParams"); + job.setCommand(new Gson().toJson(jobContent)); + } + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJobTuning.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJobTuning.java new file mode 100644 index 000000000..5f76cd64b --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/linkisjob/LinkisJobTuning.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.linkisjob; + +public interface LinkisJobTuning { + + LinkisJob tuningJob(LinkisJob job); + + boolean ifJobCantuning(String nodeType); +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectCreationOperation.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectCreationOperation.java new file mode 100644 index 000000000..05671d236 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectCreationOperation.java @@ -0,0 +1,126 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.operation; + +import com.webank.wedatasphere.dss.appconn.schedulis.ref.SchedulisProjectResponseRef; +import com.webank.wedatasphere.dss.appconn.schedulis.service.SchedulisProjectService; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.AzkabanUtils; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.SSORequestWTSS; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.SchedulisExceptionUtils; +import com.webank.wedatasphere.dss.standard.app.structure.StructureService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectCreationOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.map.ObjectMapper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.HashMap; +import java.util.Map; + +public class SchedulisProjectCreationOperation implements ProjectCreationOperation { + + private static final Logger LOGGER = LoggerFactory.getLogger(SchedulisProjectCreationOperation.class); + + private ProjectService schedulisProjectService; + + private String projectUrl; + + private String managerUrl; + + private static Long DEFAULT_PROJECT_ID = 0L; + + + public SchedulisProjectCreationOperation() { + } + + @Override + public void init() { + this.projectUrl = this.schedulisProjectService.getAppInstance().getBaseUrl().endsWith("/") ? + this.schedulisProjectService.getAppInstance().getBaseUrl() + "manager" : + this.schedulisProjectService.getAppInstance().getBaseUrl() + "/manager"; + managerUrl = this.schedulisProjectService.getAppInstance().getBaseUrl().endsWith("/") ? this.schedulisProjectService.getAppInstance().getBaseUrl() + "manager" : + this.schedulisProjectService.getAppInstance().getBaseUrl() + "/manager"; + } + + + @Override + public ProjectResponseRef createProject(ProjectRequestRef requestRef) throws ExternalOperationFailedException { + LOGGER.info("begin to create project in schedulis project is {}", requestRef.getName()); + SchedulisProjectResponseRef responseRef = new SchedulisProjectResponseRef(); + Map params = new HashMap<>(); + params.put("action", "create"); + params.put("name", requestRef.getName()); + params.put("description", requestRef.getDescription()); + try { + + String entStr = SSORequestWTSS.requestWTSSWithSSOPost(projectUrl,params,this.schedulisProjectService,requestRef.getWorkspace()); + LOGGER.error("新建工程 {}, azkaban 返回的信息是 {}", requestRef.getName(), entStr); + String message = AzkabanUtils.handleAzkabanEntity(entStr); + if (!"success".equals(message)) { + throw new ExternalOperationFailedException(90008, "新建工程失败, 原因:" + message); + } + + } catch (final Exception t) { + LOGGER.error("Failed to create project!",t); + } + try { + DEFAULT_PROJECT_ID = getSchedulisProjectId(requestRef.getName(),requestRef); + } catch (Exception e) { + SchedulisExceptionUtils.dealErrorException(60051, "failed to get project id", e, + ExternalOperationFailedException.class); + } + + responseRef.setProjectRefId(DEFAULT_PROJECT_ID); + // There is no project ID returned in schedulis, so there is no need to set. + // Other exceptions are thrown out, so the correct code returned as 0 is OK. + return responseRef; + } + + @Override + public void setStructureService(StructureService service) { + this.schedulisProjectService = (SchedulisProjectService) service; + } + + /** + * Get project ID. + */ + public Long getSchedulisProjectId(String projectName, ProjectRequestRef requestRef) throws Exception { + + Map params = new HashMap<>(); + params.put("ajax", "getProjectId"); + params.put("project", projectName); + + long projectId = 0L; + try { + String content = SSORequestWTSS.requestWTSSWithSSOGet(this.managerUrl, params, this.schedulisProjectService.getSSORequestService(), requestRef.getWorkspace()); + LOGGER.info("Get schedulis project id return str is " + content); + ObjectMapper objectMapper = new ObjectMapper(); + JsonNode jsonNode = objectMapper.readValue(content, JsonNode.class); + projectId = jsonNode.get("projectId").getLongValue(); + } catch (final Throwable t) { + SchedulisExceptionUtils.dealErrorException(60051, "failed to create project in schedulis", t, + ExternalOperationFailedException.class); + } + return projectId; + } + + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectDeletionOperation.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectDeletionOperation.java new file mode 100644 index 000000000..0163655b8 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectDeletionOperation.java @@ -0,0 +1,70 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.operation; + +import com.webank.wedatasphere.dss.appconn.schedulis.ref.SchedulisProjectResponseRef; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.SSORequestWTSS; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.SchedulisExceptionUtils; +import com.webank.wedatasphere.dss.standard.app.structure.StructureService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import java.util.HashMap; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class SchedulisProjectDeletionOperation implements ProjectDeletionOperation { + + private static final Logger LOGGER = LoggerFactory.getLogger(SchedulisProjectDeletionOperation.class); + + private ProjectService schedulisProjectService; + private String managerUrl; + + public SchedulisProjectDeletionOperation(){ + } + + @Override + public void init() { + managerUrl = this.schedulisProjectService.getAppInstance().getBaseUrl().endsWith("/") ? this.schedulisProjectService.getAppInstance().getBaseUrl() + "manager" : + this.schedulisProjectService.getAppInstance().getBaseUrl() + "/manager"; + } + + @Override + public void setStructureService(StructureService service) { + schedulisProjectService = (ProjectService) service; + init(); + } + + @Override + public ProjectResponseRef deleteProject(ProjectRequestRef projectRef) throws ExternalOperationFailedException { + try { + Map params = new HashMap<>(); + params.put("project", projectRef.getName()); + params.put("delete", "true"); + String responseContent =SSORequestWTSS.requestWTSSWithSSOGet(this.managerUrl,params,this.schedulisProjectService.getSSORequestService(),projectRef.getWorkspace()); + LOGGER.info(" deleteWtssProject --response-->{}",responseContent); + } catch (Exception e){ + SchedulisExceptionUtils.dealErrorException(60052, "failed to delete project in schedulis", e, + ExternalOperationFailedException.class); + } + return new SchedulisProjectResponseRef(); + } + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectUpdateOperation.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectUpdateOperation.java new file mode 100644 index 000000000..5ce09355c --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/operation/SchedulisProjectUpdateOperation.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.operation; + +import com.webank.wedatasphere.dss.appconn.schedulis.service.SchedulisProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.StructureService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectUpdateOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class SchedulisProjectUpdateOperation implements ProjectUpdateOperation { + + + private static final Logger LOGGER = LoggerFactory.getLogger(SchedulisProjectUpdateOperation.class); + + private ProjectService schedulisProjectService; + + public SchedulisProjectUpdateOperation(){ + + } + + @Override + public void init() { + } + + @Override + public void setStructureService(StructureService service) { + this.schedulisProjectService = (SchedulisProjectService) service; + } + + @Override + public ProjectResponseRef updateProject(ProjectRequestRef projectRef) { + return null; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/parser/AzkabanWorkflowParser.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/parser/AzkabanWorkflowParser.java new file mode 100644 index 000000000..0c0377c1f --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/parser/AzkabanWorkflowParser.java @@ -0,0 +1,94 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.parser; + +import com.google.gson.JsonObject; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanWorkflow; +import com.webank.wedatasphere.dss.common.entity.node.DSSNodeDefault; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNodeEdge; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNodeImpl; +import com.webank.wedatasphere.dss.workflow.core.json2flow.parser.WorkflowParser; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.apache.commons.beanutils.BeanUtils; + +public class AzkabanWorkflowParser implements WorkflowParser { + + @Override + public Workflow parse(JsonObject flowJson, Workflow workflow) { + AzkabanWorkflow azkabanWorkflow = new AzkabanWorkflow(); + try { + BeanUtils.copyProperties(azkabanWorkflow, workflow); + } catch (Exception e) { + throw new DSSRuntimeException(91500, "Copy workflow fields failed!", e); + } + return addEndNodeForFlowName(azkabanWorkflow); + } + + private AzkabanWorkflow addEndNodeForFlowName(AzkabanWorkflow flow) { + DSSNodeDefault endNode = new DSSNodeDefault(); + List endNodeList = getFlowEndJobList(flow); + if(flow.getRootFlow()){ + endNode.setId(flow.getName()); + endNode.setName(flow.getName()); + }else{ + endNode.setId(flow.getName() + "_"); + endNode.setName(flow.getName() + "_"); + } + endNode.setNodeType("linkis.control.empty"); + Map jobContentMap = new HashMap<>(); + endNode.setJobContent(jobContentMap); + if (!endNodeList.isEmpty()) { + if(endNodeList.size() == 1 ) { + if(endNodeList.get(0).getName().equals(flow.getName())){ + return flow; + } + } + endNodeList.forEach(tmpNode -> endNode.addDependency(tmpNode.getName())); + WorkflowNode azkabanSchedulerNode = new WorkflowNodeImpl(); + azkabanSchedulerNode.setDSSNode(endNode); + flow.getWorkflowNodes().add(azkabanSchedulerNode); + } + return flow; + } + + private List getFlowEndJobList(AzkabanWorkflow flow) { + List res = new ArrayList<>(); + for (WorkflowNode job : flow.getWorkflowNodes()) { + int flag = 0; + for (WorkflowNodeEdge link : flow.getWorkflowNodeEdges()) { + if (job.getId().equals(link.getDSSEdge().getSource())) { + flag = 1; + } + } + if (flag == 0) { + res.add(job); + } + } + return res; + } + + @Override + public int getOrder() { + return 100; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/ref/SchedulisProjectResponseRef.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/ref/SchedulisProjectResponseRef.java new file mode 100644 index 000000000..db0f7c99f --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/ref/SchedulisProjectResponseRef.java @@ -0,0 +1,83 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.ref; + +import com.webank.wedatasphere.dss.appconn.schedulis.conf.SchedulisConf; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractResponseRef; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.HashMap; +import java.util.Map; + +public class SchedulisProjectResponseRef extends AbstractResponseRef implements ProjectResponseRef { + + + private static final Logger LOGGER = LoggerFactory.getLogger(SchedulisProjectResponseRef.class); + + private String errorMsg; + + private Long projectRefId; + + public SchedulisProjectResponseRef(){ + super("", 0); + } + + + public SchedulisProjectResponseRef(String responseBody, int status, String errorMsg) { + super(responseBody, status); + this.errorMsg = errorMsg; + } + + @SuppressWarnings("unchecked") + @Override + public Map toMap() { + try{ + return SchedulisConf.gson().fromJson(this.getResponseBody(), Map.class); + }catch(Exception e){ + LOGGER.error("failed to covert {} to a map", this.getResponseBody(), e); + return new HashMap(); + } + } + + @Override + public String getErrorMsg() { + return this.errorMsg; + } + + public void setErrorMsg(String errorMsg){ + this.errorMsg = errorMsg; + } + + + @Override + public Long getProjectRefId() { + return this.projectRefId; + } + + @Override + public Map getProjectRefIds() { + return null; + } + + public void setProjectRefId(Long projectRefId){ + this.projectRefId = projectRefId; + } + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/service/AzkabanUserService.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/service/AzkabanUserService.java new file mode 100644 index 000000000..d84241f64 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/service/AzkabanUserService.java @@ -0,0 +1,72 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.service; + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.appconn.schedulis.entity.AzkabanUserEntity; +import com.webank.wedatasphere.dss.appconn.schedulis.utils.SSORequestWTSS; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +public class AzkabanUserService { + + private static Map schedulisUserMap = new HashMap<>(); + + private static final Logger LOGGER = LoggerFactory.getLogger(AzkabanUserService.class); + private static final String ADMIN_USER = CommonVars.apply("wds.dss.schedulis.admin.user", "superadmin").getValue(); + + private static void requestUserId(String baseUrl, SSORequestService ssoRequestService, Workspace workspace) { + try { + Map params = new HashMap<>(); + params.put("page", "1"); + params.put("pageSize", "10000"); + params.put("ajax","loadSystemUserSelectData"); + baseUrl = !baseUrl.endsWith("/") ? (baseUrl + "/") : baseUrl; + String finalUrl = baseUrl + "system"; + LOGGER.info("Request User info from wtss url: "+finalUrl); + String response = SSORequestWTSS.requestWTSSWithSSOGet(finalUrl,params,ssoRequestService,workspace); + Map map = DSSCommonUtils.COMMON_GSON.fromJson(response, Map.class); + if (map.get("systemUserList") instanceof List){ + ((List) map.get("systemUserList")).forEach(e -> { + AzkabanUserEntity entity = new Gson().fromJson(e.toString(), AzkabanUserEntity.class); + schedulisUserMap.put(entity.getUsername(),entity.getId()); + }); + } + } catch (Exception e) { + LOGGER.error("get user from wtss failed。", e); + } + + } + + public static String getUserIdByName(String username, String baseUrl, SSORequestService ssoRequestService, Workspace workspace) { + if(schedulisUserMap.containsKey(username)) { + return schedulisUserMap.get(username); + }else{ + requestUserId(baseUrl,ssoRequestService,workspace); + return schedulisUserMap.get(username); + } + + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/service/SchedulisProjectService.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/service/SchedulisProjectService.java new file mode 100644 index 000000000..6325a9dbf --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/service/SchedulisProjectService.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.service; + +import com.webank.wedatasphere.dss.appconn.schedulis.operation.SchedulisProjectCreationOperation; +import com.webank.wedatasphere.dss.appconn.schedulis.operation.SchedulisProjectDeletionOperation; +import com.webank.wedatasphere.dss.appconn.schedulis.operation.SchedulisProjectUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectCreationOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectUrlOperation; + +public class SchedulisProjectService extends ProjectService { + + public SchedulisProjectService(){ + } + + @Override + protected ProjectCreationOperation createProjectCreationOperation() { + return new SchedulisProjectCreationOperation(); + } + + @Override + protected ProjectUpdateOperation createProjectUpdateOperation() { + return new SchedulisProjectUpdateOperation(); + } + + @Override + protected ProjectDeletionOperation createProjectDeletionOperation() { + return new SchedulisProjectDeletionOperation(); + } + + @Override + protected ProjectUrlOperation createProjectUrlOperation() { + return null; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisHttpGet.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisHttpGet.java new file mode 100644 index 000000000..0eb88e4e8 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisHttpGet.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.sso; + +import org.apache.http.client.methods.HttpGet; + +public class SchedulisHttpGet extends HttpGet implements UserInfo { + + private String user; + + public SchedulisHttpGet(String url, String user){ + super(url); + this.user = user; + } + @Override + public String getUser() { + return user; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisHttpPost.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisHttpPost.java new file mode 100644 index 000000000..3782e5605 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisHttpPost.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.sso; + +import org.apache.http.client.methods.HttpPost; + +public class SchedulisHttpPost extends HttpPost implements UserInfo{ + + private String user; + + public SchedulisHttpPost(String url, String user){ + super(url); + this.user = user; + } + + @Override + public String getUser() { + return user; + } + + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisSecurityService.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisSecurityService.java new file mode 100644 index 000000000..0bdc6cdb4 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/SchedulisSecurityService.java @@ -0,0 +1,176 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.sso; + + +import com.google.common.cache.Cache; +import com.google.common.cache.CacheBuilder; +import com.webank.wedatasphere.dss.appconn.schedulis.conf.AzkabanConf; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import org.apache.commons.io.IOUtils; +import org.apache.http.HttpEntity; +import org.apache.http.NameValuePair; +import org.apache.http.client.CookieStore; +import org.apache.http.client.entity.UrlEncodedFormEntity; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.client.protocol.HttpClientContext; +import org.apache.http.cookie.Cookie; +import org.apache.http.impl.client.BasicCookieStore; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.message.BasicNameValuePair; +import org.apache.http.util.EntityUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.Properties; +import java.util.concurrent.TimeUnit; + + +public final class SchedulisSecurityService { + + private static Logger LOGGER = LoggerFactory.getLogger(SchedulisSecurityService.class); + + + private String securityUrl = ""; + private static final String USER_NAME_KEY = "username"; + private static final String USER_TOKEN_KEY = AzkabanConf.AZKABAN_LOGIN_PWD.getValue(); + private static final String SESSION_ID_KEY = "azkaban.browser.session.id"; + private static Properties userToken; + private static final String USER_STR = "username"; + + private static final String USER_RSA = AzkabanConf.AZKABAN_RSA.getValue(); + private static final String USER_SECRET = "dss_secret"; + + private static final String CIPHER_STR = "userpwd"; + + + private static final String SUPER_USER = "dws-wtss"; + private static final String SUPER_USER_CIPHER = "WeBankBDPWTSS&DWS@2019"; + + private static final String SUPER_USER_STR = "superUser"; + private static final String SUPER_USER_CIPHER_STR = "superUserPwd"; + + + private Cache cookieCache = CacheBuilder.newBuilder() + .expireAfterAccess(30 * 60, TimeUnit.SECONDS) + .build(); + + private static SchedulisSecurityService instance; + + private SchedulisSecurityService(String baseUrl) { + this.securityUrl = baseUrl.endsWith("/") ? baseUrl + "checkin" : baseUrl + "/checkin"; + } + + public static SchedulisSecurityService getInstance(String baseUrl) { + if (null == instance) { + synchronized (SchedulisSecurityService.class) { + if (null == instance) { + instance = new SchedulisSecurityService(baseUrl); + } + } + } + return instance; + } + + static { + Utils.defaultScheduler().scheduleAtFixedRate(()->{ + LOGGER.info("开始读取用户token文件"); + Properties properties = new Properties(); + try { + properties.load(SchedulisSecurityService.class.getClassLoader().getResourceAsStream("token.properties")); + userToken = properties; + } catch (IOException e) { + LOGGER.error("读取文件失败:",e); + } + },0,10, TimeUnit.MINUTES); + } + + + public Cookie login(String user) throws Exception { + synchronized (user.intern()) { + Cookie session = cookieCache.getIfPresent(user); + if (session != null) { + return session; + } + Cookie newCookie = getCookie(user, getUserToken(user)); + cookieCache.put(user, newCookie); + return newCookie; + } + } + + private String getUserToken(String user) { + //直接从配置文件中读取，有需求可以自己实现 + Object token = userToken.get(user); + if (token == null) { + return ""; + } + return token.toString(); + } + + + private Cookie getCookie(String user, String token) throws Exception { + HttpPost httpPost = new HttpPost(securityUrl); + List params = new ArrayList<>(); + params.add(new BasicNameValuePair(USER_NAME_KEY, user)); + params.add(new BasicNameValuePair(USER_TOKEN_KEY, token)); + params.add(new BasicNameValuePair(USER_STR, user)); + params.add(new BasicNameValuePair(CIPHER_STR, token)); + + params.add(new BasicNameValuePair(SUPER_USER_STR, SUPER_USER)); + params.add(new BasicNameValuePair(SUPER_USER_CIPHER_STR, SUPER_USER_CIPHER)); + params.add(new BasicNameValuePair(USER_SECRET, USER_RSA)); + + params.add(new BasicNameValuePair("action", "login")); + httpPost.setEntity(new UrlEncodedFormEntity(params)); + CookieStore cookieStore = new BasicCookieStore(); + CloseableHttpClient httpClient = null; + CloseableHttpResponse response = null; + HttpClientContext context; + String responseContent; + try { + httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); + context = HttpClientContext.create(); + response = httpClient.execute(httpPost, context); + HttpEntity entity = response.getEntity(); + responseContent = EntityUtils.toString(entity, "utf-8"); + LOGGER.info("Get azkaban response code is " + response.getStatusLine().getStatusCode() + ",response: " + responseContent); + if (response.getStatusLine().getStatusCode() != 200) { + throw new ErrorException(90041, responseContent); + } + + } finally { + IOUtils.closeQuietly(response); + IOUtils.closeQuietly(httpClient); + } + List cookies = context.getCookieStore().getCookies(); + Optional cookie = cookies.stream().filter(this::findSessionId).findAny(); + return cookie.orElseThrow(() -> new ErrorException(90041, "Get azkaban session is null : " + responseContent)); + } + + private boolean findSessionId(Cookie cookie) { + return SESSION_ID_KEY.equals(cookie.getName()); + } + + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/UserInfo.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/UserInfo.java new file mode 100644 index 000000000..e9285a86c --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/sso/UserInfo.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.sso; + + +public interface UserInfo { + + String getUser(); + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/standard/SchedulisStructureStandard.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/standard/SchedulisStructureStandard.java new file mode 100644 index 000000000..e28702086 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/standard/SchedulisStructureStandard.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.standard; + +import com.webank.wedatasphere.dss.appconn.schedulis.service.SchedulisProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.AbstractStructureIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; + + +/** + * Schedulis's engineering integration specification is a singleton. + */ +public class SchedulisStructureStandard extends AbstractStructureIntegrationStandard { + + private volatile static SchedulisStructureStandard instance; + + private SchedulisStructureStandard(){ + } + + public static SchedulisStructureStandard getInstance(){ + if(instance == null){ + synchronized (SchedulisStructureStandard.class){ + if (instance == null){ + instance = new SchedulisStructureStandard(); + } + } + } + return instance; + } + + @Override + protected ProjectService createProjectService() { + return new SchedulisProjectService(); + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/AzkabanUtils.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/AzkabanUtils.java new file mode 100644 index 000000000..2ecbe9b2d --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/AzkabanUtils.java @@ -0,0 +1,66 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.utils; + +import com.google.gson.Gson; +import org.apache.commons.lang.StringUtils; + +import java.io.IOException; +import java.util.Map; + +public class AzkabanUtils { + public static String handleAzkabanEntity(String entityString) throws IOException { + Gson gson = new Gson(); + Object object = gson.fromJson(entityString, Object.class); + String status = null; + String message = null; + if (object instanceof Map) { + Map map = (Map) object; + if (map.get("status") != null) { + status = map.get("status").toString(); + } + if (StringUtils.isNotEmpty(status)) { + if (null != map.get("message")) { + message = map.get("message").toString(); + } + } + if ("error".equalsIgnoreCase(status)) { + return message; + } + } + return "success"; + } + + public static String getValueFromEntity(String entityString, String searchKey) throws IOException { + Gson gson = new Gson(); + Object object = gson.fromJson(entityString, Object.class); + String status = null; + String valueStr = null; + if (object instanceof Map) { + Map map = (Map) object; + if (map.get("status") != null) { + status = map.get("status").toString(); + } + if (StringUtils.isNotEmpty(status) && status.equals("success")) { + if (null != map.get(searchKey)) { + valueStr = map.get(searchKey).toString(); + } + } + } + return valueStr; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/AzkabanUtilsScala.scala b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/AzkabanUtilsScala.scala new file mode 100644 index 000000000..7bff6817f --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/AzkabanUtilsScala.scala @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.utils + +import java.util +import scala.collection.JavaConversions._ +import scala.collection.JavaConverters._ + +object AzkabanUtilsScala { + + def getRepeatNodeName(nodeList:java.util.List[String]):util.List[String]={ + val res = nodeList.map(x=>(x,1)).groupBy(x => x._1).map(x =>(x._1,x._2.size)).filter(x=>x._2 >1).map(x=>x._1).toList + res.asJava + } + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SSORequestWTSS.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SSORequestWTSS.java new file mode 100644 index 000000000..08f99fb60 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SSORequestWTSS.java @@ -0,0 +1,162 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.utils; + +import static com.webank.wedatasphere.dss.appconn.schedulis.SchedulisAppConn.SCHEDULIS_APPCONN_NAME; + +import com.webank.wedatasphere.dss.appconn.schedulis.Action.FlowScheduleAction; +import com.webank.wedatasphere.dss.appconn.schedulis.Action.FlowScheduleGetAction; +import com.webank.wedatasphere.dss.appconn.schedulis.Action.FlowScheduleUploadAction; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppIntegrationService; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import java.util.List; +import java.util.Map; +import org.apache.commons.io.IOUtils; +import org.apache.http.Consts; +import org.apache.http.HttpEntity; +import org.apache.http.NameValuePair; +import org.apache.http.client.CookieStore; +import org.apache.http.client.entity.EntityBuilder; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.cookie.Cookie; +import org.apache.http.entity.ContentType; +import org.apache.http.impl.client.BasicCookieStore; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.protocol.HTTP; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class SSORequestWTSS { + + private final static Logger logger = LoggerFactory.getLogger(SSORequestWTSS.class); + + public static String requestWTSS(String url, String username, List params) throws Exception { + HttpPost httpPost = new HttpPost(url); + Cookie cookie = null; + CookieStore cookieStore = new BasicCookieStore(); + cookieStore.addCookie(cookie); + HttpEntity entity = EntityBuilder.create(). + setContentType(ContentType.create("application/x-www-form-urlencoded", Consts.UTF_8)) + .setParameters(params).build(); + httpPost.setEntity(entity); + CloseableHttpClient httpClient = null; + CloseableHttpResponse response = null; + try { + httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); + response = httpClient.execute(httpPost); + HttpEntity ent = response.getEntity(); + String entStr = IOUtils.toString(ent.getContent(), "utf-8"); + return entStr; + } catch (Exception e) { + logger.error("requestWTSSError-->", e); + throw e; + } finally { + IOUtils.closeQuietly(response); + IOUtils.closeQuietly(httpClient); + } + } + + + public static String requestWTSSWithSSOPost( String url, + Map params, + AppIntegrationService service, + Workspace workspace + ) throws Exception { + + try { + + FlowScheduleAction flowScheduleAction = new FlowScheduleAction(); + flowScheduleAction.getFormParams().putAll(params); + SSOUrlBuilderOperation ssoUrlBuilderOperation = workspace.getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(SCHEDULIS_APPCONN_NAME); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(workspace.getWorkspaceName()); + flowScheduleAction.setURL(ssoUrlBuilderOperation.getBuiltUrl()); + SSORequestOperation ssoRequestOperation = service.getSSORequestService().createSSORequestOperation(SCHEDULIS_APPCONN_NAME); + HttpResult previewResult = ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, flowScheduleAction); + if (previewResult.getStatusCode() == 200 || previewResult.getStatusCode() == 0) { + String response = previewResult.getResponseBody(); + return response; + }else{ + throw new ExternalOperationFailedException(50063,"User sso request failed:"+url); + } + + } catch (Exception e) { + logger.error("requestWTSSPostError-->", e); + throw e; + } + } + + public static String requestWTSSWithSSOGet( String url, + Map params, + SSORequestService service, + Workspace Workspace + ) throws Exception { + + try { + + FlowScheduleGetAction flowScheduleGetAction = new FlowScheduleGetAction(); + flowScheduleGetAction.getParameters().putAll(params); + SSOUrlBuilderOperation ssoUrlBuilderOperation =Workspace.getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(SCHEDULIS_APPCONN_NAME); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(Workspace.getWorkspaceName()); + flowScheduleGetAction.setURL(ssoUrlBuilderOperation.getBuiltUrl()); + SSORequestOperation ssoRequestOperation=service.createSSORequestOperation(SCHEDULIS_APPCONN_NAME); + HttpResult previewResult = ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, flowScheduleGetAction); + String response = previewResult.getResponseBody(); + return response; + + } catch (Exception e) { + logger.error("requestWTSSGetError-->", e); + throw e; + } + } + + + public static HttpResult requestWTSSWithSSOUpload(String url, + FlowScheduleUploadAction uploadAction, + SSORequestService service, + Workspace Workspace) throws AppStandardErrorException { + HttpResult previewResult= null; + try { + + SSOUrlBuilderOperation ssoUrlBuilderOperation = Workspace.getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(SCHEDULIS_APPCONN_NAME); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(Workspace.getWorkspaceName()); + uploadAction.setURl(ssoUrlBuilderOperation.getBuiltUrl()); + SSORequestOperation ssoRequestOperation = service.createSSORequestOperation(SCHEDULIS_APPCONN_NAME); + previewResult = ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, uploadAction); + + }catch (AppStandardErrorException e){ + logger.error("uploadWTSSGetError-->", e); + throw e; + } + return previewResult; + } +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SchedulisExceptionUtils.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SchedulisExceptionUtils.java new file mode 100644 index 000000000..c6b66c94d --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SchedulisExceptionUtils.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.utils; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + +import java.lang.reflect.Constructor; + +public class SchedulisExceptionUtils { + + public static void dealErrorException(int errorCode, String errorDesc,Throwable throwable, + Class clazz) throws T{ + T errorException = null; + try { + Constructor constructor = clazz.getConstructor(int.class, String.class, Throwable.class); + errorException = constructor.newInstance(errorCode, errorDesc, throwable); + errorException.setErrCode(errorCode); + errorException.setDesc(errorDesc); + } catch (Exception e) { + throw new RuntimeException(String.format("failed to instance %s", clazz.getName()), e); + } + errorException.initCause(throwable); + throw errorException; + } + + + public static void dealErrorException(int errorCode, String errorDesc, + Class clazz) throws T{ + T errorException = null; + try { + Constructor constructor = clazz.getConstructor(int.class, String.class); + errorException = constructor.newInstance(errorCode, errorDesc); + errorException.setErrCode(errorCode); + errorException.setDesc(errorDesc); + } catch (Exception e) { + throw new RuntimeException(String.format("failed to instance %s", clazz.getName()), e); + } + throw errorException; + } + + + + + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SchedulisUtils.java b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SchedulisUtils.java new file mode 100644 index 000000000..3f2030319 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/schedulis/utils/SchedulisUtils.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.utils; + +import com.google.gson.Gson; +import org.apache.commons.lang.StringUtils; + +import java.util.Map; + +public class SchedulisUtils { + + + public static String handleSchedulisResponse(String response){ + Gson gson = new Gson(); + Object object = gson.fromJson(response, Object.class); + String status = null; + String message = null; + if (object instanceof Map) { + Map map = (Map) object; + if (map.get("status") != null) { + status = map.get("status").toString(); + } + if (StringUtils.isNotEmpty(status)) { + if (null != map.get("message")) { + message = map.get("message").toString(); + } + } + if ("error".equalsIgnoreCase(status)) { + return message; + } + } + return "success"; + } + +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/appconn.properties b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/appconn.properties new file mode 100644 index 000000000..19365e2b5 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/appconn.properties @@ -0,0 +1,20 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + + + + + diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/init.sql b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/init.sql new file mode 100644 index 000000000..7a3b8ba17 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/init.sql @@ -0,0 +1,19 @@ +select @dss_appconn_schedulisId:=id from `dss_appconn` where `appconn_name` = 'schedulis'; + +delete from `dss_appconn_instance` where `appconn_id`=@dss_appconn_schedulisId; +delete from `dss_appconn` where `appconn_name`='schedulis'; + +INSERT INTO `dss_appconn` (`appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) +VALUES ('schedulis', 0, 1, NULL, 0, NULL, 'com.webank.wedatasphere.dss.appconn.schedulis.SchedulisAppConn', 'DSS_INSTALL_HOME_VAL/dss-appconns/schedulis/lib', ''); + +select @dss_appconn_schedulisId:=id from `dss_appconn` where `appconn_name` = 'schedulis'; +insert into `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) values(@dss_appconn_schedulisId,'DEV','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/','','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/'); + +delete from `dss_application` WHERE `name` ='schedulis'; +INSERT INTO `dss_application`(`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) +VALUES ('schedulis','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT',0,1,NULL,0,'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/manager?project=${projectName}','{\"scheduleHistory\":\"http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/manager?project=${projectName}&flow=${flowName}&hideHead=true#executions\"}',1,'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/homepage','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/api/v1/redirect'); + +select @dss_schedulis_applicationId:=id from `dss_application` WHERE `name` ='schedulis'; +delete from `dss_onestop_menu_application` WHERE title_en='Schedulis'; +INSERT INTO `dss_onestop_menu_application` (`application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) +VALUES(@dss_schedulis_applicationId,'3','Schedulis','Schedulis','Description for Schedulis.','Schedulis描述','scheduling, workflow','调度,工作流','1','enter Schedulis','进入Schedulis','user manual','用户手册','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','diaoduxitong-logo',NULL,NULL,NULL,NULL,NULL,'diaoduxitong-icon'); diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/log4j.properties b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/log4j.properties new file mode 100644 index 000000000..ee8619595 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/log4j2.xml b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/log4j2.xml new file mode 100644 index 000000000..8c40a73e8 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/resources/log4j2.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/schedulis/conf/SchedulisConf.scala b/dss-appconn/appconns/dss-schedulis-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/schedulis/conf/SchedulisConf.scala new file mode 100644 index 000000000..d54d7edc4 --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/schedulis/conf/SchedulisConf.scala @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.conf + +import java.lang +import java.lang.reflect.Type + +import com.google.gson.{Gson, GsonBuilder, JsonElement, JsonPrimitive, JsonSerializationContext, JsonSerializer} + +object SchedulisConf { + implicit val gson:Gson = new GsonBuilder().setPrettyPrinting().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").serializeNulls + .registerTypeAdapter(classOf[java.lang.Double], new JsonSerializer[java.lang.Double] { + override def serialize(t: lang.Double, `type`: Type, jsonSerializationContext: JsonSerializationContext): JsonElement = + if(t == t.longValue()) new JsonPrimitive(t.longValue()) else new JsonPrimitive(t) + }).create +} diff --git a/dss-appconn/appconns/dss-schedulis-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/schedulis/http/SchedulisHttpAction.scala b/dss-appconn/appconns/dss-schedulis-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/schedulis/http/SchedulisHttpAction.scala new file mode 100644 index 000000000..41dd239ff --- /dev/null +++ b/dss-appconn/appconns/dss-schedulis-appconn/src/main/scala/com/webank/wedatasphere/dss/appconn/schedulis/http/SchedulisHttpAction.scala @@ -0,0 +1,85 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.schedulis.http + +import java.io.{File, InputStream} +import java.util + +import com.webank.wedatasphere.dss.appconn.schedulis.conf.SchedulisConf +import com.webank.wedatasphere.linkis.httpclient.request.{GetAction, HttpAction, POSTAction, UploadAction, UserAction} + +trait SchedulisHttpAction extends UserAction{ + + private var user:String = _ + + override def setUser(user: String): Unit = this.user = user + + override def getUser: String = this.user + +} + +abstract class SchedulisGetAction extends GetAction with SchedulisHttpAction + + +abstract class ScheudlisPostAction extends POSTAction with SchedulisHttpAction{ + + override def getRequestPayload: String = SchedulisConf.gson.toJson(getRequestPayloads) + +} + + + + +case class SchedulisUploadAction(filePaths:Array[String], + _inputStreams:util.Map[String,InputStream],uploadUrl:String) extends ScheudlisPostAction with UploadAction with SchedulisHttpAction{ + + private val streamNames = new util.HashMap[String,String] + + override val files: util.Map[String, String] = { + if (null == filePaths || filePaths.length == 0) new util.HashMap[String,String]() else{ + val map = new java.util.HashMap[String, String] + filePaths foreach { + filePath => val arr = filePath.split(File.separator) + val fileName = arr(arr.length - 1) + map.put("file", filePath) + } + map + } + } + + override def inputStreams: util.Map[String, InputStream] = _inputStreams + + override def inputStreamNames: util.Map[String, String] = streamNames + + private var _user:String = _ + + override def setUser(user: String): Unit = this._user = user + + override def getUser: String = this._user + + override def getRequestPayload: String = "" + + override def getURL: String = uploadUrl +} + +class SchedulisCreateProjectAction(url:String) extends ScheudlisPostAction{ + + override def getURL: String = url + +} + + diff --git a/dss-appconn/appconns/dss-sendemail-appconn/pom.xml b/dss-appconn/appconns/dss-sendemail-appconn/pom.xml new file mode 100644 index 000000000..88969336c --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/pom.xml @@ -0,0 +1,36 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../pom.xml + + 4.0.0 + + dss-sendemail-appconn + pom + + + sendemail-appconn-core + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/pom.xml b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/pom.xml new file mode 100644 index 000000000..29ca4f2d5 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/pom.xml @@ -0,0 +1,153 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../pom.xml + + 4.0.0 + + dss-sendemail-appconn-core + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + compile + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + compile + + + + com.webank.wedatasphere.dss + dss-development-process-standard-execution + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + true + + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + true + + + + org.springframework + spring-context-support + 5.2.5.RELEASE + + + javax.mail + mail + 1.4 + + + + com.webank.wedatasphere.linkis + linkis-cs-client + ${linkis.version} + + + org.apache.httpcomponents + httpclient + 4.5.4 + compile + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + + + + src/main/resources + + **/*.properties + **/application.yml + **/bootstrap.yml + **/log4j2.xml + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/assembly/distribution.xml new file mode 100644 index 000000000..60cac86d7 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/assembly/distribution.xml @@ -0,0 +1,64 @@ + + + + dss-sendemail-appconn + + dir + + true + sendemail + + + + + + lib + true + true + false + true + true + + + + + + ${basedir}/src/main/resources + + appconn.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + * + + 0777 + conf + unix + + + + + diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/SendEmailAppConn.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/SendEmailAppConn.java new file mode 100644 index 000000000..6d13c8156 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/SendEmailAppConn.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail; + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractAppConn; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExecutionService; +import com.webank.wedatasphere.dss.standard.app.development.service.RefExecutionService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.development.standard.OnlyExecutionDevelopmentStandard; + +public class SendEmailAppConn extends AbstractAppConn implements OnlyDevelopmentAppConn { + + private DevelopmentIntegrationStandard standard; + + @Override + protected void initialize() { + standard = new OnlyExecutionDevelopmentStandard() { + @Override + public void close() { + } + + @Override + protected RefExecutionService createRefExecutionService() { + return new AbstractRefExecutionService() { + private RefExecutionOperation refExecutionOperation = new SendEmailRefExecutionOperation(); + + @Override + public RefExecutionOperation createRefExecutionOperation() { + return refExecutionOperation; + } + }; + } + + @Override + public void init() { + + } + }; + } + + @Override + public DevelopmentIntegrationStandard getOrCreateDevelopmentStandard() { + return standard; + } +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/conf/SendEmailAppConnInstanceConfiguration.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/conf/SendEmailAppConnInstanceConfiguration.java new file mode 100644 index 000000000..e039ae37d --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/conf/SendEmailAppConnInstanceConfiguration.java @@ -0,0 +1,96 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.conf; + +import com.webank.wedatasphere.dss.appconn.sendemail.email.EmailGenerator; +import com.webank.wedatasphere.dss.appconn.sendemail.email.EmailSender; +import com.webank.wedatasphere.dss.appconn.sendemail.email.generate.MultiContentEmailGenerator; +import com.webank.wedatasphere.dss.appconn.sendemail.email.sender.SpringJavaEmailSender; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.EmailContentGenerator; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.EmailContentParser; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.generator.MultiEmailContentGenerator; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser.FileEmailContentParser$; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser.HtmlEmailContentParser$; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser.PictureEmailContentParser$; +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser.TableEmailContentParser$; +import com.webank.wedatasphere.dss.appconn.sendemail.hook.SendEmailRefExecutionHook; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import java.util.List; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class SendEmailAppConnInstanceConfiguration { + + private static final Logger logger = LoggerFactory.getLogger(SendEmailAppConnInstanceConfiguration.class); + + private static final EmailGenerator EMAIL_GENERATOR = new MultiContentEmailGenerator(); + + private static final EmailSender EMAIL_SENDER = createEmailSender(); + + private static final EmailContentGenerator[] EMAIL_CONTENT_GENERATOR = createEmailContentGenerators(); + + private static final EmailContentParser[] emailContentParsers = createEmailContentParsers(); + + private static final SendEmailRefExecutionHook[] sendEmailRefExecutionHooks = createSendEmailRefExecutionHooks(); + + private static EmailSender createEmailSender() { + EmailSender emailSender = ClassUtils.getInstanceOrDefault(EmailSender.class, new SpringJavaEmailSender()); + logger.info("Try to use {} to instance a EmailSender.", emailSender.getClass().getSimpleName()); + return emailSender; + } + + private static EmailContentGenerator[] createEmailContentGenerators() { + return new EmailContentGenerator[] {new MultiEmailContentGenerator()}; + } + + private static EmailContentParser[] createEmailContentParsers() { + return new EmailContentParser[] {FileEmailContentParser$.MODULE$, + HtmlEmailContentParser$.MODULE$, PictureEmailContentParser$.MODULE$, TableEmailContentParser$.MODULE$}; + } + + private static SendEmailRefExecutionHook[] createSendEmailRefExecutionHooks() { + List hooks = ClassUtils.getInstances(SendEmailRefExecutionHook.class); + logger.info("SendEmailRefExecutionHook list is {}.", hooks); + return hooks.toArray(new SendEmailRefExecutionHook[0]); + } + + public static EmailSender getEmailSender() { + return EMAIL_SENDER; + } + + public static void init(){ + logger.info("init SendEmailAppConnInstanceConfiguration"); + } + + public static EmailGenerator getEmailGenerator() { + return EMAIL_GENERATOR; + } + + + public static EmailContentGenerator[] getEmailContentGenerators() { + return EMAIL_CONTENT_GENERATOR; + } + + public static EmailContentParser[] getEmailContentParsers() { + return emailContentParsers; + } + + public static SendEmailRefExecutionHook[] getSendEmailRefExecutionHooks() { + return sendEmailRefExecutionHooks; + } + +} diff --git a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/Email.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/Email.java similarity index 82% rename from sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/Email.java rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/Email.java index 61e1021e3..91ec8ed78 100644 --- a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/Email.java +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/Email.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,10 +14,10 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail; -/** - * Created by shanhuang on 2019/10/12. - */ +package com.webank.wedatasphere.dss.appconn.sendemail.email; + +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.Attachment; + public interface Email { String getContent(); diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/sender/AbstractEmailSender.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/sender/AbstractEmailSender.java new file mode 100644 index 000000000..53e7f7d4a --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/sender/AbstractEmailSender.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email.sender; + +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email; +import com.webank.wedatasphere.dss.appconn.sendemail.email.EmailSender; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import scala.runtime.BoxedUnit; + +import java.util.concurrent.Future; + +public abstract class AbstractEmailSender implements EmailSender { + + @Override + public Future sendAsync(Email email) { + return Utils.defaultScheduler().submit(() -> { + send(email); + return BoxedUnit.UNIT; + }); + } + +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/sender/SpringJavaEmailSender.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/sender/SpringJavaEmailSender.java new file mode 100644 index 000000000..b36bb337d --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/email/sender/SpringJavaEmailSender.java @@ -0,0 +1,99 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email.sender; + +import com.webank.wedatasphere.dss.appconn.sendemail.conf.SendEmailAppConnConfiguration; +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email; +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.Attachment; +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException; + +import java.util.Properties; +import javax.mail.internet.MimeMessage; +import javax.mail.util.ByteArrayDataSource; + +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.mail.javamail.JavaMailSenderImpl; +import org.springframework.mail.javamail.MimeMessageHelper; + +public class SpringJavaEmailSender extends AbstractEmailSender { + + + private static final Logger logger = LoggerFactory.getLogger(SpringJavaEmailSender.class); + + private JavaMailSenderImpl javaMailSender = new JavaMailSenderImpl(); + + public SpringJavaEmailSender() { + try { + Properties prop = new Properties(); + prop.put("mail.smtp.auth", Boolean.parseBoolean(SendEmailAppConnConfiguration.EMAIL_SMTP_AUTH().getValue())); + prop.put("mail.smtp.starttls.enable", Boolean.parseBoolean(SendEmailAppConnConfiguration.EMAIL_SMTP_STARTTLS_ENABLE().getValue())); + prop.put("mail.smtp.starttls.required", Boolean.parseBoolean(SendEmailAppConnConfiguration.EMAIL_SMTP_STARTTLS_REQUIRED().getValue())); + prop.put("mail.smtp.ssl.enable", Boolean.parseBoolean(SendEmailAppConnConfiguration.EMAIL_SMTP_SSL_ENABLED().getValue())); + prop.put("mail.smtp.timeout", Integer.parseInt(SendEmailAppConnConfiguration.EMAIL_SMTP_TIMEOUT().getValue())); + javaMailSender.setJavaMailProperties(prop); + } catch (Exception e) { + logger.error("Failed to read mail properties, roll back to default values.", e); + } + } + + @Override + public void send(Email email) throws EmailSendFailedException { + logger.info("Begin to send Email({}).", email.getSubject()); + try { + javaMailSender.send(parseToMimeMessage(email)); + } catch (Exception e) { + logger.error("Send email failed: ", e); + EmailSendFailedException ex = new EmailSendFailedException(80001, "Send email failed!"); + ex.initCause(e); + throw ex; + } + logger.info("Send Email({}) succeed.", email.getSubject()); + } + + private MimeMessage parseToMimeMessage(Email email) { + MimeMessage message = javaMailSender.createMimeMessage(); + try { + MimeMessageHelper messageHelper = new MimeMessageHelper(message, true); + if (StringUtils.isBlank(email.getFrom())) { + messageHelper.setFrom(SendEmailAppConnConfiguration.DEFAULT_EMAIL_FROM().getValue()); + } else { + messageHelper.setFrom(email.getFrom()); + } + messageHelper.setSubject(email.getSubject()); + messageHelper.setTo(email.getTo()); + if (StringUtils.isNotBlank(email.getCc())) { + messageHelper.setCc(email.getCc()); + } + if (StringUtils.isNotBlank(email.getBcc())) { + messageHelper.setBcc(email.getBcc()); + } + for (Attachment attachment : email.getAttachments()) { + messageHelper.addAttachment(attachment.getName(), new ByteArrayDataSource(attachment.getBase64Str(), attachment.getMediaType())); + } + messageHelper.setText(email.getContent(), true); + } catch (Exception e) { + logger.error("Send mail failed", e); + } + return message; + } + + public JavaMailSenderImpl getJavaMailSender() { + return javaMailSender; + } +} diff --git a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContent.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContent.java similarity index 77% rename from sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContent.java rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContent.java index a024c85df..c69b0d5f0 100644 --- a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContent.java +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContent.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,11 +14,8 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail; +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent; -/** - * Created by shanhuang on 2019/10/12. - */ public interface EmailContent { T getContent(); diff --git a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/exception/EmailSendFailedException.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/exception/EmailSendFailedException.java similarity index 80% rename from sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/exception/EmailSendFailedException.java rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/exception/EmailSendFailedException.java index 2f8b225a7..1f6aa8da7 100644 --- a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/exception/EmailSendFailedException.java +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/exception/EmailSendFailedException.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,13 +14,11 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.exception; +package com.webank.wedatasphere.dss.appconn.sendemail.exception; import com.webank.wedatasphere.linkis.common.exception.ErrorException; -/** - * Created by shanhuang on 2019/10/12. - */ + public class EmailSendFailedException extends ErrorException { public EmailSendFailedException(int errCode, String desc) { super(errCode, desc); diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/EmailInfo.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/EmailInfo.java new file mode 100644 index 000000000..7060ac95d --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/EmailInfo.java @@ -0,0 +1,139 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.hook; + +import java.util.Map; + +public class EmailInfo { + private String formId; + private String user; + private String dssProject; + private String widgets; + private String cc; + private String to; + private String bcc; + private String status; + private String priority; + private String alertList; + private int alertInterval = 60; + private String requestCreatedate; + private Map widgetColumns; + + public String getFormId() { + return formId; + } + + public void setFormId(String formId) { + this.formId = formId; + } + + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + public String getDssProject() { + return dssProject; + } + + public void setDssProject(String dssProject) { + this.dssProject = dssProject; + } + + public String getWidgets() { + return widgets; + } + + public void setWidgets(String widgets) { + this.widgets = widgets; + } + + public String getCc() { + return cc; + } + + public void setCc(String cc) { + this.cc = cc; + } + + public String getTo() { + return to; + } + + public void setTo(String to) { + this.to = to; + } + + public String getBcc() { + return bcc; + } + + public void setBcc(String bcc) { + this.bcc = bcc; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public String getPriority() { + return priority; + } + + public void setPriority(String priority) { + this.priority = priority; + } + + public String getAlertList() { + return alertList; + } + + public void setAlertList(String alertList) { + this.alertList = alertList; + } + + public int getAlertInterval() { + return alertInterval; + } + + public void setAlertInterval(int alertInterval) { + this.alertInterval = alertInterval; + } + + public String getRequestCreatedate() { + return requestCreatedate; + } + + public void setRequestCreatedate(String requestCreatedate) { + this.requestCreatedate = requestCreatedate; + } + + public Map getWidgetColumns() { + return widgetColumns; + } + + public void setWidgetColumns(Map widgetColumns) { + this.widgetColumns = widgetColumns; + } +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/HttpClientUtil.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/HttpClientUtil.java new file mode 100644 index 000000000..c7dea2f3b --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/HttpClientUtil.java @@ -0,0 +1,470 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.hook; + + +import org.apache.http.Consts; +import org.apache.http.HttpEntity; +import org.apache.http.HttpResponse; +import org.apache.http.NameValuePair; +import org.apache.http.client.ClientProtocolException; +import org.apache.http.client.config.RequestConfig; +import org.apache.http.client.entity.UrlEncodedFormEntity; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.entity.ContentType; +import org.apache.http.entity.StringEntity; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; +import org.apache.http.message.BasicNameValuePair; +import org.apache.http.util.EntityUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.net.ssl.TrustManager; +import javax.net.ssl.X509TrustManager; +import java.io.IOException; +import java.io.UnsupportedEncodingException; +import java.net.SocketTimeoutException; +import java.net.URLEncoder; +import java.security.cert.CertificateException; +import java.text.SimpleDateFormat; +import java.util.*; + +@SuppressWarnings("all") +public final class HttpClientUtil { + private final static Logger logger = LoggerFactory.getLogger(HttpClientUtil.class); + public final static int connectTimeout = 5000; + private static PoolingHttpClientConnectionManager connManager = null; + private static CloseableHttpClient httpclient = null; + + private static TrustManager trustAllManager = new X509TrustManager() { + @Override + public void checkClientTrusted(java.security.cert.X509Certificate[] arg0, String arg1) + throws CertificateException { + } + @Override + public void checkServerTrusted(java.security.cert.X509Certificate[] arg0, String arg1) + throws CertificateException { + } + @Override + public java.security.cert.X509Certificate[] getAcceptedIssuers() { + return null; + } + + }; + + static { + httpclient = HttpClients.createDefault(); + } + + + /** + * + * @param url + * @param timeout + * @param headerMap + * @param paramsList + * @param encoding + * @return + */ + public static String postForm(String url, int timeout, Map headerMap, List paramsList, String encoding){ + HttpPost post = new HttpPost(url); + try { + if(headerMap != null){ + for(Map.Entry entry : headerMap.entrySet()){ + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + //post.setHeader("Content-type", "application/json"); + RequestConfig requestConfig = RequestConfig.custom() + .setSocketTimeout(timeout) + .setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout) + .setExpectContinueEnabled(false).build(); + post.setConfig(requestConfig); + + post.setEntity(new UrlEncodedFormEntity(paramsList, encoding)); + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if(entity != null){ + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if(entity != null){ + entity.getContent().close(); + } + } + } finally { + if(response != null){ + response.close(); + } + } + } catch (Exception e) { + throw new RuntimeException("invoke http post error!",e); + } finally { + post.releaseConnection(); + } + return ""; + } + + /** + * 调用saltapi时 + * + * @author: XIEJIAN948@pingan.com.cn + */ + public static String postJsonBody(String url, int timeout, Map headerMap, + String paraData, String encoding) { + + logger.info("successfully start post Json Body url{} ", url); + HttpPost post = new HttpPost(url); + try { + if (headerMap != null) { + for (Map.Entry entry : headerMap.entrySet()) { + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout).setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout).setExpectContinueEnabled(false).build(); + StringEntity jsonEntity = new StringEntity(paraData, ContentType.APPLICATION_JSON); + post.setConfig(requestConfig); + post.setEntity(jsonEntity); + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + } catch (UnsupportedEncodingException e) { + logger.error("UnsupportedEncodingException", e); + throw new RuntimeException("failed post json return blank!"); + } catch (Exception e) { + logger.error("Exception", e); + throw new RuntimeException("failed post json return blank!"); + } finally { + post.releaseConnection(); + } + logger.info("successfully end post Json Body url{} ", url); + return ""; + } + + @SuppressWarnings("deprecation") + public static String invokeGet(String url, Map params, String encode, int connectTimeout, + int soTimeout) { + String responseString = null; + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(connectTimeout) + .setConnectTimeout(connectTimeout).setConnectionRequestTimeout(connectTimeout).build(); + + StringBuilder sb = new StringBuilder(); + sb.append(url); + int i = 0; + if (params != null) { + for (Map.Entry entry : params.entrySet()) { + if (i == 0 && !url.contains("?")) { + sb.append("?"); + } else { + sb.append("&"); + } + sb.append(entry.getKey()); + sb.append("="); + String value = entry.getValue(); + try { + sb.append(URLEncoder.encode(value, "UTF-8")); + } catch (UnsupportedEncodingException e) { + logger.warn("encode http get params error, value is " + value, e); + sb.append(URLEncoder.encode(value)); + } + i++; + } + } + HttpGet get = new HttpGet(sb.toString()); + get.setConfig(requestConfig); + try { + CloseableHttpResponse response = httpclient.execute(get); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + responseString = EntityUtils.toString(entity, encode); + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } catch (Exception e) { + logger.error(String.format("[HttpUtils Get]get response error, url:%s", sb.toString()), e); + return responseString; + } finally { + if (response != null) { + response.close(); + } + } + // System.out.println(String.format("[HttpUtils Get]Debug url:%s , + // response string %s:", sb.toString(), responseString)); + } catch (SocketTimeoutException e) { + logger.error(String.format("[HttpUtils Get]invoke get timout error, url:%s", sb.toString()), e); + return responseString; + } catch (Exception e) { + logger.error(String.format("[HttpUtils Get]invoke get error, url:%s", sb.toString()), e); + } finally { + get.releaseConnection(); + } + return responseString; + } + + /** + * HTTPS请求，默认超时为5S + * + * @param reqURL + * @param params + * @return + */ + public static String connectPostHttps(String reqURL, Map params) { + + String responseContent = null; + HttpPost httpPost = new HttpPost(reqURL); + try { + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(connectTimeout) + .setConnectTimeout(connectTimeout).setConnectionRequestTimeout(connectTimeout).build(); + List formParams = new ArrayList(); + httpPost.setEntity(new UrlEncodedFormEntity(formParams, Consts.UTF_8)); + httpPost.setConfig(requestConfig); + // 绑定到请求 Entry + for (Map.Entry entry : params.entrySet()) { + formParams.add(new BasicNameValuePair(entry.getKey(), entry.getValue())); + } + CloseableHttpResponse response = httpclient.execute(httpPost); + try { + // 执行POST请求 + HttpEntity entity = response.getEntity(); // 获取响应实体 + try { + if (null != entity) { + responseContent = EntityUtils.toString(entity, Consts.UTF_8); + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + logger.info("requestURI : " + httpPost.getURI() + ", responseContent: " + responseContent); + } catch (ClientProtocolException e) { + logger.error("ClientProtocolException", e); + } catch (IOException e) { + logger.error("IOException", e); + } finally { + httpPost.releaseConnection(); + } + return responseContent; + + } + + class Test { + String v; + String k; + + public String getV() { + return v; + } + + public void setV(String v) { + this.v = v; + } + + public String getK() { + return k; + } + + public void setK(String k) { + this.k = k; + } + + } + + // 随机4位数 + public static String getRandomValue() { + String str = "0123456789"; + StringBuilder sb = new StringBuilder(4); + for (int i = 0; i < 4; i++) { + char ch = str.charAt(new Random().nextInt(str.length())); + sb.append(ch); + } + return sb.toString(); + } + + // 当前时间到秒 + public static String getTimestamp() { + + Date date = new Date(); + String timestamp = String.valueOf(date.getTime() / 1000); + return timestamp; + } + + // 当前时间到秒 + public static String getNowDate() { + Date date = new Date(); + SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss"); + return sdf.format(date); + } + + /** + * 调用saltapi时 + * + * @author: XIEJIAN948@pingan.com.cn + */ + public static String postJsonBody2(String url, int timeout, Map headerMap, + List paramsList, String encoding) { + logger.info("successfully start post Json Body url{} ", url); + HttpPost post = new HttpPost(url); + try { + if (headerMap != null) { + for (Map.Entry entry : headerMap.entrySet()) { + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout).setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout).setExpectContinueEnabled(false).build(); + post.setConfig(requestConfig); + if (paramsList.size() > 0) { + UrlEncodedFormEntity entity = new UrlEncodedFormEntity(paramsList, encoding); + post.setEntity(entity); + } + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + } catch (UnsupportedEncodingException e) { + logger.error("UnsupportedEncodingException", e); + throw new RuntimeException("failed post json return blank!"); + } catch (Exception e) { + logger.error("Exception", e); + throw new RuntimeException("failed post json return blank!"); + } finally { + post.releaseConnection(); + } + logger.info("successfully end post Json Body url{} ", url); + return ""; + } + + /** + * 调用saltapi时 + * + * @author: XIEJIAN948@pingan.com.cn + */ + public static String postJsonBody3(String url, int timeout, Map headerMap, + Map paramsList, String encoding) { + HttpPost post = new HttpPost(url); + try { + if (headerMap != null) { + for (Map.Entry entry : headerMap.entrySet()) { + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout).setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout).setExpectContinueEnabled(false).build(); + post.setConfig(requestConfig); + if (paramsList.size() > 0) { + //JSONArray jsonArray = JSONArray.fromObject(paramsList); + //post.setEntity(new StringEntity(jsonArray.get(0).toString(), encoding)); + post.setEntity(new StringEntity(null, encoding)); + //logger.info("successfully start post Json Body url{},params ", url,jsonArray.get(0).toString()); + logger.info("successfully start post Json Body url{},params ", url,null); + } + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + } catch (UnsupportedEncodingException e) { + logger.error("UnsupportedEncodingException", e); + throw new RuntimeException("failed post json return blank!"); + } catch (Exception e) { + logger.error("Exception", e); + throw new RuntimeException("failed post json return blank!"); + } finally { + post.releaseConnection(); + } + logger.info("successfully end post Json Body url{} ", url); + return ""; + } + + public static String executeGet(String url) + { + String rtnStr = ""; + HttpGet httpGet = new HttpGet(url); + try { + HttpResponse httpResponse = httpclient.execute(httpGet); + //获得返回的结果 + rtnStr = EntityUtils.toString(httpResponse.getEntity()); + } catch (IOException e) { + e.printStackTrace(); + } finally { + httpGet.releaseConnection(); + } + return rtnStr; + } + +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/HttpResponseModel.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/HttpResponseModel.java new file mode 100644 index 000000000..a6965b114 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/HttpResponseModel.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.hook; + +public abstract class HttpResponseModel { + private String method; + private int status; + private String message; + + public String getMethod() { + return method; + } + + public void setMethod(String method) { + this.method = method; + } + + public int getStatus() { + return status; + } + + public void setStatus(int status) { + this.status = status; + } + + public String getMessage() { + return message; + } + + public void setMessage(String message) { + this.message = message; + } +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/WidgetMetaData.java b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/WidgetMetaData.java new file mode 100644 index 000000000..1e268538f --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/sendemail/hook/WidgetMetaData.java @@ -0,0 +1,85 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.hook; + +import java.util.List; +import java.util.Map; + +public class WidgetMetaData extends HttpResponseModel{ + public static class Meta{ + private String name; + private String updated; + private String columns; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getUpdated() { + return updated; + } + + public void setUpdated(String updated) { + this.updated = updated; + } + + public String getColumns() { + return columns; + } + + public void setColumns(String columns) { + this.columns = columns; + } + } + + + public static class Data{ + private String projectName; + private List widgetsMetaData; + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public List getWidgetsMetaData() { + return widgetsMetaData; + } + + public void setWidgetsMetaData(List widgetsMetaData) { + this.widgetsMetaData = widgetsMetaData; + } + } + + private Data data; + + public Data getData() { + return data; + } + + public void setData(Data data) { + this.data = data; + } +} + diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/SendEmailRefExecutionOperation.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/SendEmailRefExecutionOperation.scala new file mode 100644 index 000000000..29ab4a9b2 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/SendEmailRefExecutionOperation.scala @@ -0,0 +1,104 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail + +import java.util.Properties + +import com.webank.wedatasphere.dss.appconn.sendemail.conf.SendEmailAppConnInstanceConfiguration +import com.webank.wedatasphere.dss.appconn.sendemail.email.EmailSender +import com.webank.wedatasphere.dss.appconn.sendemail.email.sender.SpringJavaEmailSender +import com.webank.wedatasphere.dss.common.utils.ClassUtils +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef +import com.webank.wedatasphere.linkis.common.exception.ErrorException +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import org.springframework.mail.javamail.JavaMailSender + +import scala.collection.JavaConversions._ + +class SendEmailRefExecutionOperation extends RefExecutionOperation with Logging { + val EMAIL_FROM_DEFAULT = "email.from.default" + val EMAIL_HOST = "email.host" + val EMAIL_USERNAME = "email.username" + val EMAIL_PASSWORD = "email.password" + val EMAIL_PORT = "email.port" + val EMAIL_PROTOCOL = "email.protocol" + + private var service:DevelopmentService = _ + + private val sendEmailAppConnHooks = SendEmailAppConnInstanceConfiguration.getSendEmailRefExecutionHooks + private val emailContentParsers = SendEmailAppConnInstanceConfiguration.getEmailContentParsers + private val emailContentGenerators = SendEmailAppConnInstanceConfiguration.getEmailContentGenerators + private val emailGenerator = SendEmailAppConnInstanceConfiguration.getEmailGenerator + private val emailSender = SendEmailAppConnInstanceConfiguration.getEmailSender + + override def execute(requestRef: ExecutionRequestRef): ResponseRef = { + val instanceConfig = this.service.getAppInstance.getConfig + val properties = new Properties() + instanceConfig.foreach { + case (key: String, value: Object) => + properties.put(key, value.toString) + } + val springJavaEmailSender = new SpringJavaEmailSender() + val javaMailSender = springJavaEmailSender.getJavaMailSender + javaMailSender.setHost(properties.getProperty(EMAIL_HOST)) + javaMailSender.setPort(Integer.parseInt(properties.getProperty(EMAIL_PORT))) + javaMailSender.setUsername(properties.getProperty(EMAIL_USERNAME)) + javaMailSender.setPassword(properties.getProperty(EMAIL_PASSWORD)) + javaMailSender.setProtocol(properties.getProperty(EMAIL_PROTOCOL)) + val emailSender = ClassUtils.getInstanceOrDefault(classOf[EmailSender],springJavaEmailSender) + val response = new CompletedExecutionResponseRef(200) + val email = Utils.tryCatch { + sendEmailAppConnHooks.foreach(_.preGenerate(requestRef)) + val email = emailGenerator.generateEmail(requestRef) + emailContentParsers.foreach{ + p => Utils.tryQuietly(p.parse(email)) + } + emailContentGenerators.foreach{ + g => Utils.tryQuietly(g.generate(email)) + } + sendEmailAppConnHooks.foreach(_.preSend(requestRef, email)) + email + }{ t => + putErrorMsg("解析邮件内容失败！", t, response) + return response + } + Utils.tryCatch { + emailSender.send(email) + response.setIsSucceed(true) + }(putErrorMsg("发送邮件失败！", _, response)) + response + } + + protected def putErrorMsg(errorMsg: String, t: Throwable, + response: CompletedExecutionResponseRef): Unit = t match { + case t: Throwable => + response.setErrorMsg(errorMsg) + val exception = new ErrorException(80079, "failed to sendEmail") + exception.initCause(t) + logger.error(s"failed to send email, $errorMsg ", t) + response.setException(exception) + response.setIsSucceed(false) + } + + override def setDevelopmentService(service: DevelopmentService): Unit = { + this.service = service + } +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/conf/SendEmailAppConnConfiguration.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/conf/SendEmailAppConnConfiguration.scala new file mode 100644 index 000000000..ecabe6235 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/conf/SendEmailAppConnConfiguration.scala @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.conf + +import com.webank.wedatasphere.linkis.common.conf.CommonVars + +object SendEmailAppConnConfiguration { + + val EMAIL_IMAGE_HEIGHT = CommonVars("wds.dss.appconn.email.image.height", 500) + val EMAIL_IMAGE_WIDTH = CommonVars("wds.dss.appconn.email.image.width", 1920) + val DEFAULT_EMAIL_FROM = CommonVars("wds.dss.appconn.email.from.default", "") + val DEFAULT_EMAIL_SUFFIX = CommonVars("wds.dss.appconn.email.suffix.default", "@webank.com") + + val DEV_CHECK = CommonVars("wds.dss.appconn.email.dev.check", true) + val EMAIL_HOST = CommonVars("wds.dss.appconn.email.host", "") + val EMAIL_PORT = CommonVars("wds.dss.appconn.email.port", "") + val EMAIL_PROTOCOL = CommonVars("wds.dss.appconn.email.protocol", "smtp") + val EMAIL_USERNAME = CommonVars("wds.dss.appconn.email.username", "") + val EMAIL_PASSWORD = CommonVars("wds.dss.appconn.email.password", "") + + val EMAIL_SMTP_AUTH = CommonVars("wds.dss.appconn.email.smtp.auth", "true") + val EMAIL_SMTP_STARTTLS_ENABLE = CommonVars("wds.dss.appconn.email.smtp.starttls.enable", "true") + val EMAIL_SMTP_STARTTLS_REQUIRED = CommonVars("wds.dss.appconn.email.smtp.starttls.required", "true") + val EMAIL_SMTP_SSL_ENABLED = CommonVars("wds.dss.appconn.email.smtp.ssl.enable", "true") + val EMAIL_SMTP_TIMEOUT = CommonVars("wds.dss.appconn.email.smtp.timeout", "25000") + +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/cs/EmailCSHelper.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/cs/EmailCSHelper.scala new file mode 100644 index 000000000..6e882eb53 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/cs/EmailCSHelper.scala @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.cs + +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext +import com.webank.wedatasphere.linkis.common.utils.Logging +import com.webank.wedatasphere.linkis.cs.client.service.LinkisJobDataServiceImpl +import com.webank.wedatasphere.linkis.cs.client.utils.{ContextServiceUtils, SerializeHelper} +import com.webank.wedatasphere.linkis.cs.common.entity.enumeration.{ContextScope, ContextType} +import com.webank.wedatasphere.linkis.cs.common.entity.source.CommonContextKey +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils +import com.webank.wedatasphere.linkis.server.JSONUtils + +import scala.collection.JavaConversions._ + + +object EmailCSHelper extends Logging{ + + /** + * update by peaceWong form cs to get job ID + */ + def getJobIds(refContext: ExecutionRequestRefContext): Array[Long] = { + val contextIDStr = ContextServiceUtils.getContextIDStrByMap(refContext.getRuntimeMap) + val nodeIDs = refContext.getRuntimeMap.get("content") match { + case string: String => JSONUtils.gson.fromJson(string, classOf[java.util.List[String]]) + case list: java.util.List[String] => list + } + if (null == nodeIDs || nodeIDs.length < 1){ + throw new EmailSendFailedException(80003 ,"empty result set is not allowed") + } + info(s"From cs to getJob ids $nodeIDs.") + val jobIds = nodeIDs.map(ContextServiceUtils.getNodeNameByNodeID(contextIDStr, _)).map{ nodeName => + val contextKey = new CommonContextKey + contextKey.setContextScope(ContextScope.PUBLIC) + contextKey.setContextType(ContextType.DATA) + contextKey.setKey(CSCommonUtils.NODE_PREFIX + nodeName + CSCommonUtils.JOB_ID) + LinkisJobDataServiceImpl.getInstance().getLinkisJobData(contextIDStr, SerializeHelper.serializeContextKey(contextKey)) + }.map(_.getJobID).toArray + if (null == jobIds || jobIds.length < 1){ + throw new EmailSendFailedException(80003 ,"empty result set is not allowed") + } + info(s"Job IDs is ${jobIds.toList}.") + jobIds + } +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/EmailGenerator.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/EmailGenerator.scala new file mode 100644 index 000000000..37fccbc9d --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/EmailGenerator.scala @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef + +trait EmailGenerator { + + def generateEmail(requestRef: ExecutionRequestRef): Email + +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/EmailSender.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/EmailSender.scala new file mode 100644 index 000000000..053076442 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/EmailSender.scala @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email + +import java.util.concurrent.Future + +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException + +trait EmailSender { + + @throws(classOf[EmailSendFailedException]) + def send(email: Email): Unit + + def sendAsync(email: Email): Future[Unit] + +} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/AbstractEmail.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/AbstractEmail.scala similarity index 83% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/AbstractEmail.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/AbstractEmail.scala index 277100807..0f5b9f14e 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/AbstractEmail.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/AbstractEmail.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,16 +14,12 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.email +package com.webank.wedatasphere.dss.appconn.sendemail.email.domain -import com.webank.wedatasphere.dss.appjoint.sendemail.{Attachment, Email} -import com.webank.wedatasphere.dss.appjoint.sendemail.{Attachment, Email} +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email import scala.collection.mutable.ArrayBuffer -/** - * Created by shanhuang on 2019/10/12. - */ class AbstractEmail extends Email { private var content: String = _ diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/Attachment.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/Attachment.scala similarity index 78% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/Attachment.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/Attachment.scala index 5b9c16e6b..dbea0da83 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/Attachment.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/Attachment.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,18 +14,13 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail +package com.webank.wedatasphere.dss.appconn.sendemail.email.domain import java.io.File -/** - * Created by shanhuang on 2019/10/12. - */ trait Attachment { - def getName: String def getBase64Str: String def getFile: File def getMediaType: String - } diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/MultiContentEmail.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/MultiContentEmail.scala new file mode 100644 index 000000000..957de390a --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/MultiContentEmail.scala @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email.domain + +import java.util + +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.EmailContent + +import scala.collection.JavaConversions._ + +class MultiContentEmail extends AbstractEmail { + + private val emailContents = new util.ArrayList[EmailContent[_]]() + + def addEmailContent(emailContent: EmailContent[_]): Unit = emailContents.add(emailContent) + + def getEmailContents: Array[EmailContent[_]] = emailContents.toIterator.toArray + +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/PngAttachment.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/PngAttachment.scala new file mode 100644 index 000000000..369a55cb0 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/domain/PngAttachment.scala @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email.domain + +import java.io.File + +class PngAttachment(name: String, b64: String) extends Attachment { + + override def getName: String = name + + override def getBase64Str: String = b64 + + override def getFile: File = null //TODO write b64 to file + + override def getMediaType: String = "image/png" + +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/generate/AbstractEmailGenerator.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/generate/AbstractEmailGenerator.scala new file mode 100644 index 000000000..5d1c6b990 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/generate/AbstractEmailGenerator.scala @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email.generate + +import com.webank.wedatasphere.dss.appconn.sendemail.email.{Email, EmailGenerator} +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.AbstractEmail +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef +import com.webank.wedatasphere.linkis.common.utils.Logging + +trait AbstractEmailGenerator extends EmailGenerator with Logging{ + + protected def createEmail(): AbstractEmail + + override def generateEmail(requestRef: ExecutionRequestRef): Email = { + val email = createEmail() + generateEmailInfo(requestRef, email) + generateEmailContent(requestRef, email) + email + } + + protected def getRuntimeMap(requestRef: ExecutionRequestRef): java.util.Map[String, AnyRef] = + requestRef match { + case r: AsyncExecutionRequestRef => r.getExecutionRequestRefContext.getRuntimeMap + case _ => requestRef.getParameters + } + + protected def getExecutionRequestRefContext(requestRef: ExecutionRequestRef): ExecutionRequestRefContext = + requestRef match { + case r: AsyncExecutionRequestRef => r.getExecutionRequestRefContext + case _ => throw new EmailSendFailedException(80002, "ExecutionRequestRefContext is empty!") + } + + protected def generateEmailInfo(requestRef: ExecutionRequestRef, email: AbstractEmail): Unit = { + import scala.collection.JavaConversions._ + val runtimeMap = getRuntimeMap(requestRef) + runtimeMap foreach { + case (k, v) => logger.info(s"K is $k, V is $v") + } + val subject = if (runtimeMap.get("subject") != null) runtimeMap.get("subject").toString else "This is an email" + email.setSubject(subject) + val bcc = if (runtimeMap.get("bcc") != null) runtimeMap.get("bcc").toString else "" + email.setBcc(bcc) + val cc = if (runtimeMap.get("cc") != null) runtimeMap.get("cc").toString else "" + email.setCc(cc) + val from = if (runtimeMap.get("from") != null) runtimeMap.get("from").toString else + if(runtimeMap.get("wds.dss.workflow.submit.user") != null){ + runtimeMap.get("wds.dss.workflow.submit.user").toString + } else runtimeMap.get("user").toString + email.setFrom(from) + val to = if (runtimeMap.get("to") != null) runtimeMap.get("to").toString else "" + email.setTo(to) + } + + protected def generateEmailContent(requestRef: ExecutionRequestRef, email: AbstractEmail): Unit + +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/generate/MultiContentEmailGenerator.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/generate/MultiContentEmailGenerator.scala new file mode 100644 index 000000000..4d79dd015 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/email/generate/MultiContentEmailGenerator.scala @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.email.generate + +import com.webank.wedatasphere.dss.appconn.sendemail.cs.EmailCSHelper +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.{AbstractEmail, MultiContentEmail} +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.PictureEmailContent +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef +import com.webank.wedatasphere.linkis.storage.resultset.ResultSetFactory + +class MultiContentEmailGenerator extends AbstractEmailGenerator { + + override protected def createEmail(): AbstractEmail = new MultiContentEmail + + override protected def generateEmailContent(requestRef: ExecutionRequestRef, email: AbstractEmail): Unit = email match { + case multiContentEmail: MultiContentEmail => + val runtimeMap = getRuntimeMap(requestRef) + val refContext = getExecutionRequestRefContext(requestRef) + runtimeMap.get("category") match { + case "node" => + val resultSetFactory = ResultSetFactory.getInstance + EmailCSHelper.getJobIds(refContext).foreach { jobId => + refContext.fetchLinkisJobResultSetPaths(jobId).foreach { fsPath => + val resultSet = resultSetFactory.getResultSetByPath(fsPath) + val emailContent = resultSet.resultSetType() match { + case ResultSetFactory.PICTURE_TYPE => new PictureEmailContent(fsPath) + case ResultSetFactory.HTML_TYPE => throw new EmailSendFailedException(80003 ,"html result set is not allowed")//new HtmlEmailContent(fsPath) + case ResultSetFactory.TABLE_TYPE => throw new EmailSendFailedException(80003 ,"table result set is not allowed")//new TableEmailContent(fsPath) + case ResultSetFactory.TEXT_TYPE => throw new EmailSendFailedException(80003 ,"text result set is not allowed")//new FileEmailContent(fsPath) + } + multiContentEmail.addEmailContent(emailContent) + } + } + case "file" => throw new EmailSendFailedException(80003 ,"file content is not allowed") //addContentEmail(c => new FileEmailContent(new FsPath(c))) + case "text" => throw new EmailSendFailedException(80003 ,"text content is not allowed")//addContentEmail(new TextEmailContent(_)) + case "link" => throw new EmailSendFailedException(80003 ,"link content is not allowed")//addContentEmail(new UrlEmailContent(_)) + } + } + + + + +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContentGenerator.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContentGenerator.scala new file mode 100644 index 000000000..a6235abd2 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContentGenerator.scala @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent + +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email + +trait EmailContentGenerator { + + def generate(email: Email): Unit + +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContentParser.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContentParser.scala new file mode 100644 index 000000000..a85d52269 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/EmailContentParser.scala @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent + +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email + +trait EmailContentParser { + + def parse(email: Email): Unit + +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/ArrayEmailContent.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/ArrayEmailContent.scala new file mode 100644 index 000000000..2fc552dd8 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/ArrayEmailContent.scala @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain + +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.EmailContent + +class ArrayEmailContent extends EmailContent[Array[String]] { + + private var content: Array[String] = _ + + override def getContent: Array[String] = content + + override def setContent(content: Array[String]): Unit = this.content = content + +} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/FsPathStoreEmailContent.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/FsPathStoreEmailContent.scala similarity index 79% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/FsPathStoreEmailContent.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/FsPathStoreEmailContent.scala index 059912155..c392ec3fb 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/FsPathStoreEmailContent.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/FsPathStoreEmailContent.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,13 +14,10 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain import com.webank.wedatasphere.linkis.common.io.FsPath -/** - * Created by enjoyyin on 2019/10/13. - */ trait FsPathStoreEmailContent { private var fsPath: FsPath = _ diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/StringEmailContent.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/StringEmailContent.scala new file mode 100644 index 000000000..e28ab0bc8 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/StringEmailContent.scala @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain + +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.EmailContent + +class StringEmailContent extends EmailContent[String] { + + private var content: String = _ + + override def getContent: String = content + + override def setContent(content: String): Unit = this.content = content + +} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/package.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/package.scala similarity index 85% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/package.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/package.scala index 525a1409e..91feb6317 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/package.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/domain/package.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,16 +14,12 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.StringEmailContent import com.webank.wedatasphere.linkis.common.io.FsPath import scala.beans.BeanProperty -/** - * Created by enjoyyin on 2019/10/13. - */ package object emailcontent { } diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/generator/AbstractEmailContentGenerator.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/generator/AbstractEmailContentGenerator.scala similarity index 75% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/generator/AbstractEmailContentGenerator.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/generator/AbstractEmailContentGenerator.scala index 53cf5a0fb..388a36d1c 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/generator/AbstractEmailContentGenerator.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/generator/AbstractEmailContentGenerator.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,23 +14,17 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.generator +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.generator import java.text.SimpleDateFormat import java.util.{Calendar, Date} -import com.webank.wedatasphere.dss.appjoint.sendemail.{Email, EmailContentGenerator} -import com.webank.wedatasphere.dss.appjoint.sendemail.{Email, EmailContentGenerator} +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.EmailContentGenerator + -/** - * Created by shanhuang on 2019/10/12. - */ trait AbstractEmailContentGenerator extends EmailContentGenerator { - /** - * 兼容Visualis老版的变量设置方式 - * @param email - */ protected def formatSubjectOfOldVersion(email: Email): Unit = { var title = email.getSubject if (title.contains("YYYY-MM-DD HH:MM:SS")) { @@ -53,7 +46,6 @@ trait AbstractEmailContentGenerator extends EmailContentGenerator { val timeStr = sdf.format(new Date) title = title + timeStr } - // if (!title.contains("试运行版")) title = "【试运行版】" + title email.setSubject(title) } diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/generator/MultiEmailContentGenerator.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/generator/MultiEmailContentGenerator.scala new file mode 100644 index 000000000..4de225c6b --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/generator/MultiEmailContentGenerator.scala @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.generator + +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.MultiContentEmail +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.{ArrayEmailContent, StringEmailContent} +import com.webank.wedatasphere.linkis.common.utils.Logging + + +class MultiEmailContentGenerator extends AbstractEmailContentGenerator with Logging { + + override def generate(email: Email): Unit = email match { + case multiContentEmail: MultiContentEmail => + formatSubjectOfOldVersion(email) + formatSubject(multiContentEmail) + formatContent(multiContentEmail) + } + + protected def formatContent(email: MultiContentEmail): Unit = { + val sb: StringBuilder = new StringBuilder("") + sb.append("") + email.getEmailContents.foreach { + case emailContent: ArrayEmailContent => + emailContent.getContent.foreach(content => sb.append("")) + case emailContent: StringEmailContent => + sb.append("") + } + sb.append("
").append(content).append("
").append(emailContent.getContent).append("
") + sb.append("") + email.setContent(sb.toString) + } + +} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/AbstractEmailContentParser.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/AbstractEmailContentParser.scala similarity index 75% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/AbstractEmailContentParser.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/AbstractEmailContentParser.scala index 98249bcdf..05851d718 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/AbstractEmailContentParser.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/AbstractEmailContentParser.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,16 +14,14 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.parser +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser import java.lang.reflect.{ParameterizedType, Type} -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.FsPathStoreEmailContent -import com.webank.wedatasphere.dss.appjoint.sendemail.{Email, EmailContent, EmailContentParser} -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.FsPathStoreEmailContent -import com.webank.wedatasphere.dss.appjoint.sendemail.{Email, EmailContent, EmailContentParser} +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.MultiContentEmail +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.FsPathStoreEmailContent +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.{EmailContent, EmailContentParser} import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetReader import com.webank.wedatasphere.linkis.common.io.{MetaData, Record} import com.webank.wedatasphere.linkis.common.utils.Utils @@ -32,9 +29,6 @@ import com.webank.wedatasphere.linkis.storage.LineRecord import com.webank.wedatasphere.linkis.storage.resultset.ResultSetReader import org.apache.commons.io.IOUtils -/** - * Created by shanhuang on 2019/10/12. - */ abstract class AbstractEmailContentParser[T] extends EmailContentParser { override def parse(email: Email): Unit = email match { diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/FileEmailContentParser.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/FileEmailContentParser.scala new file mode 100644 index 000000000..91d3eefb3 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/FileEmailContentParser.scala @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser + +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.MultiContentEmail +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.FileEmailContent +import com.webank.wedatasphere.linkis.common.utils.Utils +import com.webank.wedatasphere.linkis.storage.LineRecord +import org.apache.commons.io.IOUtils + +object FileEmailContentParser extends AbstractEmailContentParser[FileEmailContent] { + override protected def parseEmailContent(emailContent: FileEmailContent, + multiContentEmail: MultiContentEmail): Unit = { + val reader = getResultSetReader(emailContent) + val content = new StringBuilder + Utils.tryFinally{ + while(reader.hasNext) { + reader.getRecord match { + case lineRecord: LineRecord => + content.append(lineRecord.getLine).append("
") + } + } + }(IOUtils.closeQuietly(reader)) + emailContent.setContent(content.toString()) + } +} \ No newline at end of file diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/HtmlEmailContentParser.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/HtmlEmailContentParser.scala new file mode 100644 index 000000000..cab083f70 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/HtmlEmailContentParser.scala @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser + +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.MultiContentEmail +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.HtmlEmailContent + +object HtmlEmailContentParser extends AbstractEmailContentParser[HtmlEmailContent] { + override protected def parseEmailContent(emailContent: HtmlEmailContent, + multiContentEmail: MultiContentEmail): Unit = { + getFirstLineRecord(emailContent).foreach(emailContent.setContent) + } +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/PictureEmailContentParser.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/PictureEmailContentParser.scala new file mode 100644 index 000000000..1b771409e --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/PictureEmailContentParser.scala @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser + +import java.awt.image.BufferedImage +import java.io.{ByteArrayInputStream, ByteArrayOutputStream} +import java.util.{Base64, UUID} + +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.{AbstractEmail, MultiContentEmail, PngAttachment} +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.PictureEmailContent +import com.webank.wedatasphere.linkis.common.conf.Configuration +import javax.imageio.ImageIO +import org.apache.commons.codec.binary.Base64OutputStream +import com.webank.wedatasphere.dss.appconn.sendemail.conf.SendEmailAppConnConfiguration._ + +object PictureEmailContentParser extends AbstractEmailContentParser[PictureEmailContent] { + + override protected def parseEmailContent(emailContent: PictureEmailContent, + multiContentEmail: MultiContentEmail): Unit = { + getFirstLineRecord(emailContent).foreach { imageStr => + val decoder = Base64.getDecoder + val byteArr = decoder.decode(imageStr) + val inputStream = new ByteArrayInputStream(byteArr) + val image = ImageIO.read(inputStream) + val contents = generateImage(image, multiContentEmail) + emailContent.setContent(contents) + } + } + + protected def generateImage(bufferedImage: BufferedImage, email: AbstractEmail): Array[String] = { + val imageUUID: String = UUID.randomUUID.toString + val width: Int = bufferedImage.getWidth + val height: Int = bufferedImage.getHeight + val imagesCuts = if (height > EMAIL_IMAGE_HEIGHT.getValue) { + val numOfCut = Math.ceil(height.toDouble / EMAIL_IMAGE_HEIGHT.getValue).toInt + val realHeight = height / numOfCut + (0 until numOfCut).map(i => bufferedImage.getSubimage(0, i * realHeight, width, realHeight)).toArray + } else Array(bufferedImage) + imagesCuts.indices.map { index => + val image = imagesCuts(index) + val imageName = index + "_" + imageUUID + ".png" + val os = new ByteArrayOutputStream + val b64Stream = new Base64OutputStream(os) + ImageIO.write(image, "png", b64Stream) + val b64 = os.toString(Configuration.BDP_ENCODING.getValue) + email.addAttachment(new PngAttachment(imageName, b64)) + + var iHeight = image.getHeight + var iWidth = image.getWidth + + if (iWidth > EMAIL_IMAGE_WIDTH.getValue) { + iHeight = ((EMAIL_IMAGE_WIDTH.getValue.toDouble / iWidth.toDouble) * iHeight.toDouble).toInt + iWidth = EMAIL_IMAGE_WIDTH.getValue + } + s"""""" + }.toArray + } + +} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/TableEmailContentParser.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/TableEmailContentParser.scala similarity index 78% rename from sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/TableEmailContentParser.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/TableEmailContentParser.scala index 3b832185e..fecabbe7a 100644 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/TableEmailContentParser.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/emailcontent/parser/TableEmailContentParser.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,19 +14,14 @@ * */ -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.parser +package com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.parser -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.TableEmailContent +import com.webank.wedatasphere.dss.appconn.sendemail.email.domain.MultiContentEmail +import com.webank.wedatasphere.dss.appconn.sendemail.emailcontent.domain.TableEmailContent import com.webank.wedatasphere.linkis.common.utils.Utils import com.webank.wedatasphere.linkis.storage.resultset.table.{TableMetaData, TableRecord} import org.apache.commons.io.IOUtils import org.apache.commons.lang.StringUtils -import org.springframework.stereotype.Component - -/** - * Created by shanhuang on 2019/10/12. - */ object TableEmailContentParser extends AbstractEmailContentParser[TableEmailContent] { override protected def parseEmailContent(emailContent: TableEmailContent, @@ -39,9 +33,11 @@ object TableEmailContentParser extends AbstractEmailContentParser[TableEmailCont writeTableTH(tableMetaData, content) } Utils.tryFinally { - while(reader.hasNext) reader.getRecord match { - case tableRecord: TableRecord => - writeTableTR(tableRecord, content) + while(reader.hasNext) { + reader.getRecord match { + case tableRecord: TableRecord => + writeTableTR(tableRecord, content) + } } }(IOUtils.closeQuietly(reader)) emailContent.setContent(content.toString()) diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/hook/AbstractSendEmailRefExecutionHook.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/hook/AbstractSendEmailRefExecutionHook.scala new file mode 100644 index 000000000..b146b5930 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/hook/AbstractSendEmailRefExecutionHook.scala @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.hook + +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef + +abstract class AbstractSendEmailRefExecutionHook extends SendEmailRefExecutionHook { + + protected def getExecutionRequestRefContext(requestRef: ExecutionRequestRef): ExecutionRequestRefContext = requestRef match { + case async: AsyncExecutionRequestRef => async.getExecutionRequestRefContext + case _ => throw new EmailSendFailedException(80002, "ExecutionRequestRefContext is empty!") + } + + override def preSend(requestRef: ExecutionRequestRef, email: Email): Unit = {} +} diff --git a/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/hook/SendEmailRefExecutionHook.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/hook/SendEmailRefExecutionHook.scala new file mode 100644 index 000000000..3c2bd7c00 --- /dev/null +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/dss/appconn/sendemail/hook/SendEmailRefExecutionHook.scala @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.sendemail.hook + +import com.webank.wedatasphere.dss.appconn.sendemail.email.Email +import com.webank.wedatasphere.dss.appconn.sendemail.exception.EmailSendFailedException +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef + + +trait SendEmailRefExecutionHook { + + @throws(classOf[EmailSendFailedException]) + def preGenerate(requestRef: ExecutionRequestRef): Unit + + def preSend(requestRef: ExecutionRequestRef, email: Email): Unit + +} diff --git a/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/linkis/server/JSONUtils.scala b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/linkis/server/JSONUtils.scala similarity index 83% rename from visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/linkis/server/JSONUtils.scala rename to dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/linkis/server/JSONUtils.scala index c5b348ff0..7267ae81d 100644 --- a/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/linkis/server/JSONUtils.scala +++ b/dss-appconn/appconns/dss-sendemail-appconn/sendemail-appconn-core/src/main/scala/com/webank/wedatasphere/linkis/server/JSONUtils.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,11 +16,9 @@ package com.webank.wedatasphere.linkis.server -/** - * Created by enjoyyin on 2019/10/12. - */ object JSONUtils { val gson = BDPJettyServerHelper.gson + val jackson = BDPJettyServerHelper.jacksonJson } diff --git a/dss-appconn/appconns/dss-visualis-appconn/pom.xml b/dss-appconn/appconns/dss-visualis-appconn/pom.xml new file mode 100644 index 000000000..fed552a4f --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/pom.xml @@ -0,0 +1,207 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-visualis-appconn + + + + com.webank.wedatasphere.dss + dss-project-plugin + ${dss.version} + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + com.webank.wedatasphere.dss + spring-origin-dss-project-plugin + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-structure-integration-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-development-process-standard-execution + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + httpclient + org.apache.httpcomponents + + + true + + + com.webank.wedatasphere.linkis + linkis-cs-common + ${linkis.version} + compile + + + linkis-bml-client + + + gson + com.google.code.gson + + + com.webank.wedatasphere.linkis + ${linkis.version} + provided + true + + + + com.webank.wedatasphere.linkis + linkis-httpclient + ${linkis.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + + + linkis-common + com.webank.wedatasphere.linkis + + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + org.apache.maven.plugins + maven-gpg-plugin + + true + + + + + + src/main/java + + **/*.xml + + + + src/main/resources + + **/*.properties + **/application.yml + **/bootstrap.yml + **/log4j2.xml + + + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-visualis-appconn/src/main/assembly/distribution.xml new file mode 100644 index 000000000..89f053c69 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/assembly/distribution.xml @@ -0,0 +1,74 @@ + + + + dss-visualis-appconn + + dir + + true + visualis + + + + + + lib + true + true + false + true + true + + + + + + + + ${basedir}/conf + + * + + 0777 + conf + unix + + + . + + */** + + logs + + + + ${basedir}/src/main/resources + + init.sql + + 0777 + db + + + + + + + diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisAppConn.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisAppConn.java new file mode 100644 index 000000000..976301827 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisAppConn.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis; + +import com.webank.wedatasphere.dss.appconn.core.ext.ThirdlyAppConn; +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractOnlySSOAppConn; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.structure.StructureIntegrationStandard; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +public class VisualisAppConn extends AbstractOnlySSOAppConn implements ThirdlyAppConn { + + public static final String VISUALIS_APPCONN_NAME = CommonVars.apply("wds.dss.appconn.visualis.name", "Visualis").getValue(); + + private VisualisDevelopmentIntegrationStandard developmentIntegrationStandard; + private VisualisStructureIntegrationStandard structureIntegrationStandard; + + @Override + protected void initialize() { + structureIntegrationStandard = new VisualisStructureIntegrationStandard(); + developmentIntegrationStandard = new VisualisDevelopmentIntegrationStandard(); + } + + @Override + public StructureIntegrationStandard getOrCreateStructureStandard() { + return structureIntegrationStandard; + } + + @Override + public DevelopmentIntegrationStandard getOrCreateDevelopmentStandard() { + return developmentIntegrationStandard; + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisDevelopmentIntegrationStandard.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisDevelopmentIntegrationStandard.java new file mode 100644 index 000000000..5065cb5e8 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisDevelopmentIntegrationStandard.java @@ -0,0 +1,51 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis; + +import com.webank.wedatasphere.dss.appconn.visualis.execution.VisualisExecutionService; +import com.webank.wedatasphere.dss.appconn.visualis.service.*; +import com.webank.wedatasphere.dss.standard.app.development.service.*; +import com.webank.wedatasphere.dss.standard.app.development.standard.AbstractDevelopmentIntegrationStandard; + +public class VisualisDevelopmentIntegrationStandard extends AbstractDevelopmentIntegrationStandard { + + @Override + protected RefCRUDService createRefCRUDService() { + return new VisualisCRUDService(); + } + + @Override + protected RefExecutionService createRefExecutionService() { + return new VisualisExecutionService(); + } + + @Override + protected RefExportService createRefExportService() { + return new VisualisRefExportService(); + } + + @Override + protected RefImportService createRefImportService() { + return new VisualisRefImportService(); + } + + @Override + protected RefQueryService createRefQueryService() { + return new VisualisQueryService(); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisStructureIntegrationStandard.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisStructureIntegrationStandard.java new file mode 100644 index 000000000..c35fb1357 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/VisualisStructureIntegrationStandard.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis; + +import com.webank.wedatasphere.dss.appconn.visualis.project.VisualisProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.AbstractStructureIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; + + +public class VisualisStructureIntegrationStandard extends AbstractStructureIntegrationStandard { + + @Override + protected ProjectService createProjectService() { + return new VisualisProjectService(); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisCompletedExecutionResponseRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisCompletedExecutionResponseRef.java new file mode 100644 index 000000000..6442fca49 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisCompletedExecutionResponseRef.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.execution; + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; + +import java.util.Map; + +public class VisualisCompletedExecutionResponseRef extends CompletedExecutionResponseRef { + + public VisualisCompletedExecutionResponseRef(int status, String errorMessage){ + super(status); + this.errorMsg = errorMessage; + } + + public VisualisCompletedExecutionResponseRef(int status) { + super(status); + } + + public VisualisCompletedExecutionResponseRef(String responseBody, int status) { + super(responseBody, status); + } + + @Override + public Map toMap() { + return null; + } + + @Override + public String getErrorMsg() { + return this.errorMsg; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisExecutionService.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisExecutionService.java new file mode 100644 index 000000000..0970daec6 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisExecutionService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.execution; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExecutionService; + +public class VisualisExecutionService extends AbstractRefExecutionService { + + @Override + public RefExecutionOperation createRefExecutionOperation() { + VisualisRefExecutionOperation visualisRefExecutionOperation = new VisualisRefExecutionOperation(this); + return visualisRefExecutionOperation; + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisRefExecutionOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisRefExecutionOperation.java new file mode 100644 index 000000000..0b0d1337c --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/execution/VisualisRefExecutionOperation.java @@ -0,0 +1,212 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.execution; + +import com.google.common.collect.Lists; +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.WidgetResultData; +import com.webank.wedatasphere.dss.appconn.visualis.ref.VisualisCommonResponseRef; +import com.webank.wedatasphere.dss.appconn.visualis.utils.NumberUtils; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.appconn.visualis.utils.VisualisDownloadAction; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.SSOIntegrationConf; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetWriter; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration; +import com.webank.wedatasphere.linkis.storage.LineMetaData; +import com.webank.wedatasphere.linkis.storage.LineRecord; +import com.webank.wedatasphere.linkis.storage.domain.Column; +import com.webank.wedatasphere.linkis.storage.domain.DataType; +import com.webank.wedatasphere.linkis.storage.resultset.table.TableMetaData; +import com.webank.wedatasphere.linkis.storage.resultset.table.TableRecord; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.ByteArrayOutputStream; +import java.util.Base64; +import java.util.List; +import java.util.Map; + +public class VisualisRefExecutionOperation implements RefExecutionOperation { + + private final static Logger logger = LoggerFactory.getLogger(VisualisRefExecutionOperation.class); + DevelopmentService developmentService; + private SSORequestOperation ssoRequestOperation; + + public VisualisRefExecutionOperation(DevelopmentService service) { + this.developmentService = service; + this.ssoRequestOperation = this.developmentService.getSSORequestService().createSSORequestOperation(getAppName()); + } + + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public ResponseRef execute(ExecutionRequestRef ref) throws ExternalOperationFailedException { + AsyncExecutionRequestRef asyncExecutionRequestRef = (AsyncExecutionRequestRef) ref; + String nodeType = asyncExecutionRequestRef.getExecutionRequestRefContext().getRuntimeMap().get("nodeType").toString(); + if("visualis.widget".equalsIgnoreCase(nodeType)){ + return executeWidget(asyncExecutionRequestRef); + } else if("visualis.display".equalsIgnoreCase(nodeType)){ + return executePreview(asyncExecutionRequestRef, + URLUtils.getUrl(getBaseUrl(), URLUtils.DISPLAY_PREVIEW_URL_FORMAT, getId(asyncExecutionRequestRef)), + URLUtils.getUrl(getBaseUrl(), URLUtils.DISPLAY_METADATA_URL_FORMAT, getId(asyncExecutionRequestRef))); + } else if("visualis.dashboard".equalsIgnoreCase(nodeType)){ + return executePreview(asyncExecutionRequestRef, + URLUtils.getUrl(getBaseUrl(), URLUtils.DASHBOARD_PREVIEW_URL_FORMAT, getId(asyncExecutionRequestRef)), + URLUtils.getUrl(getBaseUrl(), URLUtils.DASHBOARD_METADATA_URL_FORMAT, getId(asyncExecutionRequestRef))); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + nodeType, null); + } + } + + + private ResponseRef executeWidget(AsyncExecutionRequestRef ref) throws ExternalOperationFailedException { + String url = URLUtils.getUrl(getBaseUrl(), URLUtils.WIDGET_DATA_URL_FORMAT, getId(ref)); + ref.getExecutionRequestRefContext().appendLog("Ready to get result set from " + url); + List columns = Lists.newArrayList(); + List tableRecords = Lists.newArrayList(); + VisualisDownloadAction visualisDownloadAction = new VisualisDownloadAction(); + visualisDownloadAction.setUser(getUser(ref)); + SSOUrlBuilderOperation ssoUrlBuilderOperation = ref.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(ref.getWorkspace().getWorkspaceName()); + try{ + visualisDownloadAction.setURL(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisDownloadAction); + WidgetResultData responseData = BDPJettyServerHelper.gson().fromJson(IOUtils.toString(visualisDownloadAction.getInputStream()), WidgetResultData.class); + if(responseData.getData().getColumns().isEmpty()){ + ref.getExecutionRequestRefContext().appendLog("Cannot execute an empty Widget!"); + throw new ExternalOperationFailedException(90176, "Cannot execute an empty Widget!", null); + } + for (WidgetResultData.Column columnData : responseData.getData().getColumns()) { + columns.add(new Column(columnData.getName(), DataType.toDataType(columnData.getType().toLowerCase()), "")); + } + ResultSetWriter resultSetWriter = ref.getExecutionRequestRefContext().createTableResultSetWriter(); + resultSetWriter.addMetaData(new TableMetaData(columns.toArray(new Column[0]))); + for (Map recordMap : responseData.getData().getResultList()) { + resultSetWriter.addRecord(new TableRecord(recordMap.values().toArray())); + } + resultSetWriter.flush(); + IOUtils.closeQuietly(resultSetWriter); + ref.getExecutionRequestRefContext().sendResultSet(resultSetWriter); + } catch (Throwable e){ + ref.getExecutionRequestRefContext().appendLog("Failed to debug Widget url " + url); + ref.getExecutionRequestRefContext().appendLog(e.getMessage()); + logger.error("executeWidget error:",e); + throw new ExternalOperationFailedException(90176, "Failed to debug Widget", e); + } finally { + IOUtils.closeQuietly(visualisDownloadAction.getInputStream()); + } + return new VisualisCompletedExecutionResponseRef(200); + } + + private ResponseRef executePreview(AsyncExecutionRequestRef ref, String previewUrl, String metaUrl) throws ExternalOperationFailedException { + ref.getExecutionRequestRefContext().appendLog("Ready to get result set from " + previewUrl); + VisualisDownloadAction previewDownloadAction = new VisualisDownloadAction(); + previewDownloadAction.setUser(getUser(ref)); + + VisualisDownloadAction metadataDownloadAction = new VisualisDownloadAction(); + metadataDownloadAction.setUser(getUser(ref)); + + try{ + logger.info("got workspace" + ref.getWorkspace()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = ref.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(previewUrl); + ssoUrlBuilderOperation.setWorkspace(ref.getWorkspace().getWorkspaceName()); + logger.info("got getSSOUrlBuilderOperation:" + SSOIntegrationConf.gson().toJson(ssoUrlBuilderOperation)); + logger.info("got getSSOUrlBuilderOperation built url:" + ssoUrlBuilderOperation.getBuiltUrl()); + previewDownloadAction.setURL(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult previewResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, previewDownloadAction); + ByteArrayOutputStream os = new ByteArrayOutputStream(); + IOUtils.copy(previewDownloadAction.getInputStream(), os); + String response = new String(Base64.getEncoder().encode(os.toByteArray())); + + SSOUrlBuilderOperation ssoUrlBuilderOperationMeta = ref.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperationMeta.setAppName(getAppName()); + ssoUrlBuilderOperationMeta.setAppName(getAppName()); + ssoUrlBuilderOperationMeta.setReqUrl(metaUrl); + ssoUrlBuilderOperationMeta.setWorkspace(ref.getWorkspace().getWorkspaceName()); + metadataDownloadAction.setURL(ssoUrlBuilderOperationMeta.getBuiltUrl()); + HttpResult metaResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperationMeta, metadataDownloadAction); + String metadata = StringUtils.chomp(IOUtils.toString(metadataDownloadAction.getInputStream(), ServerConfiguration.BDP_SERVER_ENCODING().getValue())); + ResultSetWriter resultSetWriter = ref.getExecutionRequestRefContext().createPictureResultSetWriter(); + resultSetWriter.addMetaData(new LineMetaData(metadata)); + resultSetWriter.addRecord(new LineRecord(response)); + resultSetWriter.flush(); + IOUtils.closeQuietly(resultSetWriter); + ref.getExecutionRequestRefContext().sendResultSet(resultSetWriter); + } catch (Throwable e){ + ref.getExecutionRequestRefContext().appendLog("Failed to debug Display url " + previewUrl); + logger.error(e.getMessage(), e); + throw new ExternalOperationFailedException(90176, "Failed to debug Display", e); + } finally { + IOUtils.closeQuietly(previewDownloadAction.getInputStream()); + IOUtils.closeQuietly(metadataDownloadAction.getInputStream()); + } + return new VisualisCompletedExecutionResponseRef(200); + } + + private String getUser(AsyncExecutionRequestRef requestRef) { + return requestRef.getExecutionRequestRefContext().getRuntimeMap().get("wds.dss.workflow.submit.user").toString(); + } + + private String getId(AsyncExecutionRequestRef requestRef) { + try { + String executionContent = BDPJettyServerHelper.jacksonJson().writeValueAsString(requestRef.getJobContent()); + String nodeType = requestRef.getExecutionRequestRefContext().getRuntimeMap().get("nodeType").toString(); + if("visualis.display".equalsIgnoreCase(nodeType)){ + VisualisCommonResponseRef displayCreateResponseRef = new VisualisCommonResponseRef(executionContent); + return NumberUtils.parseDoubleString(displayCreateResponseRef.getDisplayId()); + } else if("visualis.dashboard".equalsIgnoreCase(nodeType)){ + VisualisCommonResponseRef dashboardCreateResponseRef = new VisualisCommonResponseRef(executionContent); + return NumberUtils.parseDoubleString(dashboardCreateResponseRef.getDashboardId()); + } else if ("visualis.widget".equalsIgnoreCase(nodeType)){ + VisualisCommonResponseRef widgetCreateResponseRef = new VisualisCommonResponseRef(executionContent); + return NumberUtils.parseDoubleString(widgetCreateResponseRef.getWidgetId()); + } + } catch (Exception e) { + logger.error("failed to parse jobContent", e); + } + return null; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + private String getBaseUrl(){ + return developmentService.getAppInstance().getBaseUrl(); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/HttpResponseModel.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/HttpResponseModel.java new file mode 100644 index 000000000..15e7a7baa --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/HttpResponseModel.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.model; + +public abstract class HttpResponseModel { + private String method; + private int status; + private String message; + + public String getMethod() { + return method; + } + + public void setMethod(String method) { + this.method = method; + } + + public int getStatus() { + return status; + } + + public void setStatus(int status) { + this.status = status; + } + + public String getMessage() { + return message; + } + + public void setMessage(String message) { + this.message = message; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisDeleteAction.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisDeleteAction.java new file mode 100644 index 000000000..2755d9a2e --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisDeleteAction.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.model; + +import com.webank.wedatasphere.linkis.httpclient.request.DeleteAction; +import com.webank.wedatasphere.linkis.httpclient.request.UserAction; + +public class VisualisDeleteAction extends DeleteAction implements UserAction { + + String url; + String user; + + @Override + public String getURL() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + @Override + public String getUser() { + return user; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisGetAction.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisGetAction.java new file mode 100644 index 000000000..bc22aaa25 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisGetAction.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.model; + +import com.webank.wedatasphere.linkis.httpclient.request.GetAction; +import com.webank.wedatasphere.linkis.httpclient.request.UserAction; + +public class VisualisGetAction extends GetAction implements UserAction { + + String url; + String user; + + @Override + public String getURL() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + @Override + public String getUser() { + return user; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisPostAction.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisPostAction.java new file mode 100644 index 000000000..fdc7c65b6 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisPostAction.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.model; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction; +import com.webank.wedatasphere.linkis.httpclient.request.UserAction; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class VisualisPostAction extends POSTAction implements UserAction { + + private static final Logger LOGGER = LoggerFactory.getLogger(VisualisPostAction.class); + String url; + String user; + + @Override + public String getRequestPayload() { + try { + return BDPJettyServerHelper.jacksonJson().writeValueAsString(getRequestPayloads()); + } catch (JsonProcessingException e) { + LOGGER.error("failed to covert {} to a string", getRequestPayloads(), e); + return ""; + } + } + + @Override + public String getURL() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + @Override + public String getUser() { + return user; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisPutAction.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisPutAction.java new file mode 100644 index 000000000..40ee9bee7 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/VisualisPutAction.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.model; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.webank.wedatasphere.linkis.httpclient.request.PutAction; +import com.webank.wedatasphere.linkis.httpclient.request.UserAction; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class VisualisPutAction extends PutAction implements UserAction { + + private static final Logger LOGGER = LoggerFactory.getLogger(VisualisPutAction.class); + String url; + String user; + + @Override + public String getRequestPayload() { + try { + return BDPJettyServerHelper.jacksonJson().writeValueAsString(getRequestPayloads()); + } catch (JsonProcessingException e) { + LOGGER.error("failed to covert {} to a string", getRequestPayloads(), e); + return ""; + } + } + + @Override + public String getURL() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + @Override + public String getUser() { + return user; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/WidgetResultData.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/WidgetResultData.java new file mode 100644 index 000000000..405c69eec --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/model/WidgetResultData.java @@ -0,0 +1,76 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.model; + +import java.util.List; +import java.util.Map; + +public class WidgetResultData extends HttpResponseModel{ + public static class Column{ + private String name; + private String type; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + } + + + public static class Data{ + private List columns; + private List> resultList; + + public List getColumns() { + return columns; + } + + public void setColumns(List columns) { + this.columns = columns; + } + + public List> getResultList() { + return resultList; + } + + public void setResultList(List> resultList) { + this.resultList = resultList; + } + } + + private Data data; + + public Data getData() { + return data; + } + + public void setData(Data data) { + this.data = data; + } +} + diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefCreationOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefCreationOperation.java new file mode 100644 index 000000000..97b7f8d0e --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefCreationOperation.java @@ -0,0 +1,249 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.operation; + +import com.google.common.collect.Maps; +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.google.gson.reflect.TypeToken; +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisPostAction; +import com.webank.wedatasphere.dss.appconn.visualis.ref.*; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.entity.node.DSSNodeDefault; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.appconn.visualis.ref.NodeUpdateCSRequestRefImpl; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.ref.NodeRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CreateRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.List; +import java.util.Map; + +public class VisualisRefCreationOperation implements RefCreationOperation { + private final static Logger logger = LoggerFactory.getLogger(VisualisRefCreationOperation.class); + + DevelopmentService developmentService; + private SSORequestOperation ssoRequestOperation; + + public VisualisRefCreationOperation(DevelopmentService service){ + this.developmentService = service; + this.ssoRequestOperation = this.developmentService.getSSORequestService().createSSORequestOperation(getAppName()); + } + + + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public ResponseRef createRef(CreateRequestRef requestRef) throws ExternalOperationFailedException { + NodeRequestRef visualisCreateRequestRef = (NodeRequestRef) requestRef; + requestRef.setParameter("projectId", visualisCreateRequestRef.getProjectId()); + if("linkis.appconn.visualis.widget".equalsIgnoreCase(visualisCreateRequestRef.getNodeType())){ + return sendWidgetRequest(visualisCreateRequestRef); + } else if("linkis.appconn.visualis.display".equalsIgnoreCase(visualisCreateRequestRef.getNodeType())){ + return sendDisplayRequest(visualisCreateRequestRef); + }else if("linkis.appconn.visualis.dashboard".equalsIgnoreCase(visualisCreateRequestRef.getNodeType())){ + return sendDashboardRequest(visualisCreateRequestRef); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + visualisCreateRequestRef.getNodeType(), null); + } + } + + private ResponseRef sendWidgetRequest(NodeRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.widgetUrl; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getUserName()); + visualisPostAction.addRequestPayload("widgetName", requestRef.getName()); + visualisPostAction.addRequestPayload("projectId", requestRef.getParameter("projectId")); + visualisPostAction.addRequestPayload(CSCommonUtils.CONTEXT_ID_STR, requestRef.getJobContent().get(CSCommonUtils.CONTEXT_ID_STR)); + if(requestRef.getJobContent().get("bindViewKey") != null){ + String viewNodeName = requestRef.getJobContent().get("bindViewKey").toString(); + if(StringUtils.isNotBlank(viewNodeName) && !"empty".equals(viewNodeName)){ + viewNodeName = getNodeNameByKey(viewNodeName,(String) requestRef.getJobContent().get("json")); + visualisPostAction.addRequestPayload(CSCommonUtils.NODE_NAME_STR, viewNodeName); + } + } + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + ResponseRef responseRef; + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + // + HttpResult httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + responseRef = new VisualisCommonResponseRef(httpResult.getResponseBody()); + } catch (Exception e){ + throw new ExternalOperationFailedException(90177, "Create Widget Exception", e); + } + if(responseRef.isFailed()){ + logger.error(responseRef.getResponseBody()); + throw new ExternalOperationFailedException(90178,responseRef.getErrorMsg()); + } + // cs + VisualisRefUpdateOperation visualisRefUpdateOperation = new VisualisRefUpdateOperation(developmentService); + NodeUpdateCSRequestRefImpl visualisUpdateCSRequestRef = new NodeUpdateCSRequestRefImpl(); + visualisUpdateCSRequestRef.setContextID((String) requestRef.getJobContent().get(CSCommonUtils.CONTEXT_ID_STR)); + visualisUpdateCSRequestRef.setJobContent(responseRef.toMap()); + visualisUpdateCSRequestRef.setUserName(requestRef.getUserName()); + visualisUpdateCSRequestRef.setNodeType(requestRef.getNodeType()); + visualisUpdateCSRequestRef.setWorkspace(requestRef.getWorkspace()); + visualisRefUpdateOperation.updateRef(visualisUpdateCSRequestRef); + return responseRef; + } + + public static String getNodeNameByKey(String key, String json) { + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(json).getAsJsonObject(); + JsonArray nodeJsonArray = jsonObject.getAsJsonArray("nodes"); + List dwsNodes = DSSCommonUtils.COMMON_GSON.fromJson(nodeJsonArray, new TypeToken>() { + }.getType()); + return dwsNodes.stream().filter(n -> key.equals(n.getId())).map(DSSNode::getName).findFirst().orElse(""); + } + + + private ResponseRef sendDisplayRequest(NodeRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.displayUrl; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getUserName()); + visualisPostAction.addRequestPayload("name", requestRef.getName()); + visualisPostAction.addRequestPayload("projectId", requestRef.getParameter("projectId")); + visualisPostAction.addRequestPayload("avatar", "18"); + visualisPostAction.addRequestPayload("publish", true); + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + VisualisCommonResponseRef responseRef; + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + responseRef = new VisualisCommonResponseRef(httpResult.getResponseBody()); + } catch (Exception e){ + throw new ExternalOperationFailedException(90177, "Create Display Exception", e); + } + createDisplaySlide(responseRef, requestRef); + return responseRef; + } + + private ResponseRef sendDashboardRequest(NodeRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.dashboardPortalUrl; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getUserName()); + visualisPostAction.addRequestPayload("name", requestRef.getName()); + visualisPostAction.addRequestPayload("projectId", requestRef.getParameter("projectId")); + visualisPostAction.addRequestPayload("avatar", "18"); + visualisPostAction.addRequestPayload("publish", true); + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + VisualisCommonResponseRef responseRef; + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + responseRef = new VisualisCommonResponseRef(httpResult.getResponseBody()); + } catch (Exception e){ + throw new ExternalOperationFailedException(90177, "Create Dashboard Exception", e); + } + createDashboard(responseRef, requestRef); + return responseRef; + } + + private void createDisplaySlide(VisualisCommonResponseRef displayCreateResponseRef, NodeRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.displayUrl + "/" + displayCreateResponseRef.getDisplayId() + "/slides"; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getUserName()); + visualisPostAction.addRequestPayload("config", URLUtils.displaySlideConfig); + visualisPostAction.addRequestPayload("displayId", Long.parseLong(displayCreateResponseRef.getDisplayId())); + visualisPostAction.addRequestPayload("index", 0); + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + Map resMap = Maps.newHashMap(); + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + resMap = BDPJettyServerHelper.jacksonJson().readValue(httpResult.getResponseBody(), Map.class); + } catch (Exception e){ + throw new ExternalOperationFailedException(90177, "Create DisplaySlide Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90176, errorMsg, null); + } + } + + private void createDashboard(VisualisCommonResponseRef dashboardCreateResponseRef, NodeRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.dashboardPortalUrl + "/" + dashboardCreateResponseRef.getDashboardId() + "/dashboards"; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getUserName()); + visualisPostAction.addRequestPayload("config", ""); + visualisPostAction.addRequestPayload("dashboardPortalId", Long.parseLong(dashboardCreateResponseRef.getDashboardId())); + visualisPostAction.addRequestPayload("index", 0); + visualisPostAction.addRequestPayload("name", requestRef.getName()); + visualisPostAction.addRequestPayload("parentId", 0); + visualisPostAction.addRequestPayload("type", 1); + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + Map resMap = Maps.newHashMap(); + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + resMap = BDPJettyServerHelper.jacksonJson().readValue(httpResult.getResponseBody(), Map.class); + } catch (Exception e){ + throw new ExternalOperationFailedException(90177, "Create Dashboard Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90176, errorMsg, null); + } + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + private String getBaseUrl(){ + return developmentService.getAppInstance().getBaseUrl(); + } + + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefDeletionOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefDeletionOperation.java new file mode 100644 index 000000000..f8789df36 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefDeletionOperation.java @@ -0,0 +1,168 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.operation; + +import com.google.common.collect.Maps; +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisDeleteAction; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.appconn.visualis.utils.VisualisNodeUtils; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.ref.NodeRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; + +import java.util.Map; + +public class VisualisRefDeletionOperation implements RefDeletionOperation { + + private DevelopmentService developmentService; + private SSORequestOperation ssoRequestOperation; + + public VisualisRefDeletionOperation(DevelopmentService service) { + this.developmentService = service; + this.ssoRequestOperation = this.developmentService.getSSORequestService().createSSORequestOperation(getAppName()); + } + + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public void deleteRef(RequestRef requestRef) throws ExternalOperationFailedException { + NodeRequestRef visualisDeleteRequestRef = (NodeRequestRef) requestRef; + if ("linkis.appconn.visualis.widget".equalsIgnoreCase(visualisDeleteRequestRef.getNodeType())) { + deleteWidget(visualisDeleteRequestRef); + } else if ("linkis.appconn.visualis.display".equalsIgnoreCase(visualisDeleteRequestRef.getNodeType())) { + deleteDisplay(visualisDeleteRequestRef); + } else if ("linkis.appconn.visualis.dashboard".equalsIgnoreCase(visualisDeleteRequestRef.getNodeType())) { + deleteDashboardPortal(visualisDeleteRequestRef); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + visualisDeleteRequestRef.getNodeType(), null); + } + } + + private void deleteWidget(NodeRequestRef visualisDeleteRequestRef) throws ExternalOperationFailedException { + String url = null; + try { + url = getBaseUrl() + URLUtils.widgetDeleteUrl + "/" + VisualisNodeUtils.getId(visualisDeleteRequestRef); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Delete Widget Exception", e); + } + VisualisDeleteAction deleteAction = new VisualisDeleteAction(); + deleteAction.setUser(visualisDeleteRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisDeleteRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisDeleteRequestRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + deleteAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, deleteAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Delete Widget Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90177, errorMsg, null); + } + } + + private void deleteDisplay(NodeRequestRef visualisDeleteRequestRef) throws ExternalOperationFailedException { + String url = null; + try { + url = getBaseUrl() + URLUtils.displayUrl + "/" + VisualisNodeUtils.getId(visualisDeleteRequestRef); + } catch (Exception e) { + new ExternalOperationFailedException(90177, "Delete Display Exception", e); + } + VisualisDeleteAction deleteAction = new VisualisDeleteAction(); + deleteAction.setUser(visualisDeleteRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisDeleteRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisDeleteRequestRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + deleteAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, deleteAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Delete Display Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90177, errorMsg, null); + } + } + + private void deleteDashboardPortal(NodeRequestRef visualisDeleteRequestRef) throws ExternalOperationFailedException { + String url = null; + try { + url = getBaseUrl() + URLUtils.dashboardPortalUrl + "/" + VisualisNodeUtils.getId(visualisDeleteRequestRef); + } catch (Exception e) { + new ExternalOperationFailedException(90177, "Delete Dashboard Exception", e); + } + VisualisDeleteAction deleteAction = new VisualisDeleteAction(); + deleteAction.setUser(visualisDeleteRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisDeleteRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisDeleteRequestRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + deleteAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = (HttpResult) this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, deleteAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Delete Display Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90177, errorMsg, null); + } + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + private String getBaseUrl() { + return developmentService.getAppInstance().getBaseUrl(); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefExportOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefExportOperation.java new file mode 100644 index 000000000..5ded49d2f --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefExportOperation.java @@ -0,0 +1,103 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.operation; + +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisPostAction; +import com.webank.wedatasphere.dss.appconn.visualis.publish.VisualisExportResponseRef; +import com.webank.wedatasphere.dss.appconn.visualis.ref.VisualisCommonResponseRef; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.ref.ExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class VisualisRefExportOperation implements RefExportOperation { + + private final static Logger logger = LoggerFactory.getLogger(VisualisRefExportOperation.class); + + DevelopmentService developmentService; + private SSORequestOperation ssoRequestOperation; + + public VisualisRefExportOperation(DevelopmentService developmentService){ + this.developmentService = developmentService; + this.ssoRequestOperation = this.developmentService.getSSORequestService().createSSORequestOperation(getAppName()); + } + + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public ResponseRef exportRef(ExportRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.projectUrl + "/export"; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getParameter("user").toString()); + visualisPostAction.addRequestPayload("projectId", requestRef.getParameter("projectId")); + visualisPostAction.addRequestPayload("partial", true); + String nodeType = requestRef.getParameter("nodeType").toString(); + String externalContent = null; + try { + externalContent = BDPJettyServerHelper.jacksonJson().writeValueAsString(requestRef.getParameter("jobContent")); + if("linkis.appconn.visualis.widget".equalsIgnoreCase(nodeType)){ + VisualisCommonResponseRef widgetCreateResponseRef = new VisualisCommonResponseRef(externalContent); + visualisPostAction.addRequestPayload("widgetIds", ((Double) Double.parseDouble(widgetCreateResponseRef.getWidgetId())).longValue()); + } else if("linkis.appconn.visualis.display".equalsIgnoreCase(nodeType)){ + VisualisCommonResponseRef displayCreateResponseRef = new VisualisCommonResponseRef(externalContent); + visualisPostAction.addRequestPayload("displayIds", ((Double) Double.parseDouble(displayCreateResponseRef.getDisplayId())).longValue()); + } else if("linkis.appconn.visualis.dashboard".equalsIgnoreCase(nodeType)){ + VisualisCommonResponseRef dashboardCreateResponseRef = new VisualisCommonResponseRef(externalContent); + visualisPostAction.addRequestPayload("dashboardPortalIds", ((Double) Double.parseDouble(dashboardCreateResponseRef.getDashboardId())).longValue()); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + requestRef.getType(), null); + } + } catch (Exception e) { + logger.error("Failed to create export request", e); + } + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + ResponseRef responseRef; + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + responseRef = new VisualisExportResponseRef(httpResult.getResponseBody()); + } catch (Exception e){ + throw new ExternalOperationFailedException(90176, "Export Visualis Exception", e); + } + return responseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + developmentService = service; + } + + private String getBaseUrl(){ + return developmentService.getAppInstance().getBaseUrl(); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefImportOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefImportOperation.java new file mode 100644 index 000000000..2901bd375 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefImportOperation.java @@ -0,0 +1,87 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.operation; + +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisPostAction; +import com.webank.wedatasphere.dss.appconn.visualis.publish.VisualisImportResponseRef; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.ref.ImportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Map; + +public class VisualisRefImportOperation implements RefImportOperation { + + private final static Logger logger = LoggerFactory.getLogger(VisualisRefImportOperation.class); + + DevelopmentService developmentService; + private SSORequestOperation ssoRequestOperation; + + public VisualisRefImportOperation(DevelopmentService developmentService){ + this.developmentService = developmentService; + this.ssoRequestOperation = this.developmentService.getSSORequestService().createSSORequestOperation(getAppName()); + } + + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public ResponseRef importRef(ImportRequestRef requestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.projectUrl + "/import"; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(requestRef.getParameter("user").toString()); + visualisPostAction.addRequestPayload("projectId", requestRef.getParameter("projectId")); + visualisPostAction.addRequestPayload("projectVersion", "v1"); + visualisPostAction.addRequestPayload("flowVersion", requestRef.getParameter("orcVersion")); + visualisPostAction.addRequestPayload("resourceId", requestRef.getParameter("resourceId")); + visualisPostAction.addRequestPayload("version", requestRef.getParameter("version")); + SSOUrlBuilderOperation ssoUrlBuilderOperation = requestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(requestRef.getWorkspace().getWorkspaceName()); + ResponseRef responseRef; + try{ + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + responseRef = new VisualisImportResponseRef((Map) requestRef.getParameter("jobContent"), httpResult.getResponseBody(), requestRef.getParameter("nodeType").toString(), requestRef.getParameter("projectId")); + } catch (Exception e){ + throw new ExternalOperationFailedException(90176, "Export Visualis Exception", e); + } + return responseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + private String getBaseUrl(){ + return developmentService.getAppInstance().getBaseUrl(); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefQueryOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefQueryOperation.java new file mode 100644 index 000000000..3e028d9d9 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefQueryOperation.java @@ -0,0 +1,77 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.operation; + +import com.webank.wedatasphere.dss.appconn.visualis.ref.VisualisCommonResponseRef; +import com.webank.wedatasphere.dss.appconn.visualis.ref.VisualisOpenRequestRef; +import com.webank.wedatasphere.dss.appconn.visualis.ref.VisualisOpenResponseRef; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.OpenRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; + +import java.util.HashMap; +import java.util.Map; + +public class VisualisRefQueryOperation implements RefQueryOperation { + + DevelopmentService developmentService; + + @Override + public ResponseRef query(OpenRequestRef ref) throws ExternalOperationFailedException { + VisualisOpenRequestRef visualisOpenRequestRef = (VisualisOpenRequestRef) ref; + try { + String externalContent = BDPJettyServerHelper.jacksonJson().writeValueAsString(visualisOpenRequestRef.getJobContent()); + Long projectId = (Long) visualisOpenRequestRef.getParameter("projectId"); + String baseUrl = visualisOpenRequestRef.getParameter("redirectUrl").toString(); + String jumpUrl = baseUrl; + if("linkis.appconn.visualis.widget".equalsIgnoreCase(visualisOpenRequestRef.getType())){ + VisualisCommonResponseRef widgetCreateResponseRef = new VisualisCommonResponseRef(externalContent); + jumpUrl = URLUtils.getUrl(baseUrl, URLUtils.WIDGET_JUMP_URL_FORMAT, projectId.toString(), widgetCreateResponseRef.getWidgetId()); + } else if("linkis.appconn.visualis.display".equalsIgnoreCase(visualisOpenRequestRef.getType())){ + VisualisCommonResponseRef displayCreateResponseRef = new VisualisCommonResponseRef(externalContent); + jumpUrl = URLUtils.getUrl(baseUrl, URLUtils.DISPLAY_JUMP_URL_FORMAT, projectId.toString(), displayCreateResponseRef.getDisplayId()); + }else if("linkis.appconn.visualis.dashboard".equalsIgnoreCase(visualisOpenRequestRef.getType())){ + VisualisCommonResponseRef dashboardCreateResponseRef = new VisualisCommonResponseRef(externalContent); + jumpUrl = URLUtils.getUrl(baseUrl, URLUtils.DASHBOARD_JUMP_URL_FORMAT, projectId.toString(), dashboardCreateResponseRef.getDashboardId(), visualisOpenRequestRef.getName()); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + visualisOpenRequestRef.getType(), null); + } + String retJumpUrl = getEnvUrl(jumpUrl, visualisOpenRequestRef); + Map retMap = new HashMap<>(); + retMap.put("jumpUrl",retJumpUrl); + return new VisualisOpenResponseRef(DSSCommonUtils.COMMON_GSON.toJson(retMap), 0); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Failed to parse jobContent ", e); + } + } + + public String getEnvUrl(String url, VisualisOpenRequestRef visualisOpenRequestRef ){ + String env = ((Map) visualisOpenRequestRef.getParameter("params")).get(DSSCommonUtils.DSS_LABELS_KEY).toString(); + return url + "?env=" + env.toLowerCase(); + } + + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefUpdateOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefUpdateOperation.java new file mode 100644 index 000000000..113464bd8 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/operation/VisualisRefUpdateOperation.java @@ -0,0 +1,234 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.operation; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisPostAction; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisPutAction; +import com.webank.wedatasphere.dss.appconn.visualis.utils.URLUtils; +import com.webank.wedatasphere.dss.appconn.visualis.utils.VisualisNodeUtils; +import com.webank.wedatasphere.dss.standard.app.development.ref.UpdateCSRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.ref.NodeRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.UpdateRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; + +import java.util.Map; + +public class VisualisRefUpdateOperation implements RefUpdateOperation { + + DevelopmentService developmentService; + private SSORequestOperation ssoRequestOperation; + + public VisualisRefUpdateOperation(DevelopmentService developmentService) { + this.developmentService = developmentService; + + this.ssoRequestOperation = developmentService.getSSORequestService().createSSORequestOperation(getAppName()); + } + + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public ResponseRef updateRef(UpdateRequestRef requestRef) throws ExternalOperationFailedException { + if (!(requestRef instanceof UpdateCSRequestRef)) { + NodeRequestRef visualisUpdateRequestRef = (NodeRequestRef) requestRef; + if ("linkis.appconn.visualis.widget".equalsIgnoreCase(visualisUpdateRequestRef.getNodeType())) { + return updateWidget(visualisUpdateRequestRef); + } else if ("linkis.appconn.visualis.display".equalsIgnoreCase(visualisUpdateRequestRef.getNodeType())) { + return updateDisplay(visualisUpdateRequestRef); + } else if ("linkis.appconn.visualis.dashboard".equalsIgnoreCase(visualisUpdateRequestRef.getNodeType())) { + return updateDashboardPortal(visualisUpdateRequestRef); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + visualisUpdateRequestRef.getNodeType(), null); + } + } else { + NodeRequestRef visualisUpdateCSRequestRef = (NodeRequestRef) requestRef; + if ("linkis.appconn.visualis.widget".equalsIgnoreCase(visualisUpdateCSRequestRef.getNodeType())) { + return updateWidgetCS(visualisUpdateCSRequestRef); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + visualisUpdateCSRequestRef.getNodeType(), null); + } + } + + } + + private ResponseRef updateWidgetCS(NodeRequestRef visualisUpdateCSRequestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.widgetContextUrl; + VisualisPostAction postAction = new VisualisPostAction(); + try { + postAction.addRequestPayload("id", Integer.parseInt(VisualisNodeUtils.getId(visualisUpdateCSRequestRef))); + postAction.addRequestPayload(CSCommonUtils.CONTEXT_ID_STR, visualisUpdateCSRequestRef.getContextID()); + postAction.setUser(visualisUpdateCSRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisUpdateCSRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisUpdateCSRequestRef.getWorkspace().getWorkspaceName()); + postAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + HttpResult httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, postAction); + String response = httpResult.getResponseBody(); + Map resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + int status = (int) resMap.get("status"); + if (status != 0) { + String errorMsg = resMap.get("message").toString(); + throw new ExternalOperationFailedException(90177, errorMsg); + } + return new CommonResponseRef(); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Update CS Exception", e); + } + } + + private ResponseRef updateDashboardPortal(NodeRequestRef visualisUpdateRequestRef) throws ExternalOperationFailedException { + String url = null; + String id = null; + try { + id = VisualisNodeUtils.getId(visualisUpdateRequestRef); + url = getBaseUrl() + URLUtils.dashboardPortalUrl + "/" + id; + } catch (Exception e) { + new ExternalOperationFailedException(90177, "Update Dashboard Exception", e); + } + VisualisPutAction putAction = new VisualisPutAction(); + putAction.addRequestPayload("projectId", visualisUpdateRequestRef.getProjectId()); + putAction.addRequestPayload("name", visualisUpdateRequestRef.getName()); + putAction.addRequestPayload("id", Long.parseLong(id)); + putAction.addRequestPayload("avatar", "9"); + putAction.addRequestPayload("description", ""); + putAction.addRequestPayload("publish", true); + putAction.addRequestPayload("roleIds", Lists.newArrayList()); + putAction.setUser(visualisUpdateRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisUpdateRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisUpdateRequestRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + putAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, putAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Update Dashboard Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90177, errorMsg, null); + } + return new CommonResponseRef(); + } + + private ResponseRef updateDisplay(NodeRequestRef visualisUpdateRequestRef) throws ExternalOperationFailedException { + String url = null; + String id = null; + try { + id = VisualisNodeUtils.getId(visualisUpdateRequestRef); + url = getBaseUrl() + URLUtils.displayUrl + "/" + id; + } catch (Exception e) { + new ExternalOperationFailedException(90177, "Update Display Exception", e); + } + + VisualisPutAction putAction = new VisualisPutAction(); + putAction.addRequestPayload("projectId", visualisUpdateRequestRef.getProjectId()); + putAction.addRequestPayload("name", visualisUpdateRequestRef.getName()); + putAction.addRequestPayload("id", Long.parseLong(id)); + putAction.addRequestPayload("avatar", "9"); + putAction.addRequestPayload("description", ""); + putAction.addRequestPayload("publish", true); + putAction.addRequestPayload("roleIds", Lists.newArrayList()); + putAction.setUser(visualisUpdateRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisUpdateRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisUpdateRequestRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + putAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, putAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Update Display Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + if (code != 200) { + String errorMsg = header.toString(); + throw new ExternalOperationFailedException(90177, errorMsg, null); + } + return new CommonResponseRef(); + } + + private ResponseRef updateWidget(NodeRequestRef visualisUpdateRequestRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + URLUtils.widgetUpdateUrl; + VisualisPostAction postAction = new VisualisPostAction(); + try { + postAction.addRequestPayload("id", Long.parseLong(VisualisNodeUtils.getId(visualisUpdateRequestRef))); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Update Widget Exception", e); + } + postAction.addRequestPayload("name", visualisUpdateRequestRef.getName()); + postAction.setUser(visualisUpdateRequestRef.getUserName()); + SSOUrlBuilderOperation ssoUrlBuilderOperation = visualisUpdateRequestRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(visualisUpdateRequestRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + postAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, postAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + throw new ExternalOperationFailedException(90177, "Update Widget Exception", e); + } + int status = (int) resMap.get("status"); + if (status != 0) { + String errorMsg = resMap.get("message").toString(); + throw new ExternalOperationFailedException(90177, errorMsg); + } + return new CommonResponseRef(); + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + private String getBaseUrl() { + return developmentService.getAppInstance().getBaseUrl(); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectCreationOperation.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectCreationOperation.java new file mode 100644 index 000000000..1b7166ffe --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectCreationOperation.java @@ -0,0 +1,109 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.project; + +import com.google.common.collect.Maps; +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.appconn.visualis.model.VisualisPostAction; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.app.structure.StructureService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectCreationOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class VisualisProjectCreationOperation implements ProjectCreationOperation { + + private static Logger logger = LoggerFactory.getLogger(VisualisProjectCreationOperation.class); + private final static String projectUrl = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/projects"; + private SSORequestOperation ssoRequestOperation; + private StructureService structureService; + + public VisualisProjectCreationOperation(StructureService service, SSORequestOperation ssoRequestOperation) { + this.structureService = service; + this.ssoRequestOperation = ssoRequestOperation; + } + private String getAppName() { + return VisualisAppConn.VISUALIS_APPCONN_NAME; + } + + @Override + public void init() { + } + + @Override + public ProjectResponseRef createProject(ProjectRequestRef projectRef) throws ExternalOperationFailedException { + String url = getBaseUrl() + projectUrl; + VisualisPostAction visualisPostAction = new VisualisPostAction(); + visualisPostAction.setUser(projectRef.getCreateBy()); + visualisPostAction.addRequestPayload("name", projectRef.getName()); + visualisPostAction.addRequestPayload("description", projectRef.getDescription()); + visualisPostAction.addRequestPayload("pic", "6"); + visualisPostAction.addRequestPayload("visibility", true); + SSOUrlBuilderOperation ssoUrlBuilderOperation = projectRef.getWorkspace().getSSOUrlBuilderOperation().copy(); + ssoUrlBuilderOperation.setAppName(getAppName()); + ssoUrlBuilderOperation.setReqUrl(url); + ssoUrlBuilderOperation.setWorkspace(projectRef.getWorkspace().getWorkspaceName()); + String response = ""; + Map resMap = Maps.newHashMap(); + HttpResult httpResult = null; + try { + visualisPostAction.setUrl(ssoUrlBuilderOperation.getBuiltUrl()); + httpResult = this.ssoRequestOperation.requestWithSSO(ssoUrlBuilderOperation, visualisPostAction); + response = httpResult.getResponseBody(); + resMap = BDPJettyServerHelper.jacksonJson().readValue(response, Map.class); + } catch (Exception e) { + logger.error("Create Visualis Project Exception", e); + throw new ExternalOperationFailedException(90176, "Create Visualis Project Exception", e); + } + Map header = (Map) resMap.get("header"); + int code = (int) header.get("code"); + String errorMsg = ""; + if (code != 200) { + errorMsg = header.toString(); + throw new ExternalOperationFailedException(90176, errorMsg, null); + } + Integer projectId = (Integer) ((Map) resMap.get("payload")).get("id"); + VisualisProjectResponseRef visualisProjectResponseRef = null; + try { + visualisProjectResponseRef = new VisualisProjectResponseRef(response, code); + } catch (Exception e) { + throw new ExternalOperationFailedException(90176, "failed to parse response json", e); + } + visualisProjectResponseRef.setAppInstance(structureService.getAppInstance()); + visualisProjectResponseRef.setProjectRefId(projectId.longValue()); + visualisProjectResponseRef.setErrorMsg(errorMsg); + return visualisProjectResponseRef; + } + + @Override + public void setStructureService(StructureService service) { + this.structureService = service; + } + + private String getBaseUrl(){ + return structureService.getAppInstance().getBaseUrl(); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectResponseRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectResponseRef.java new file mode 100644 index 000000000..27c761b5d --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectResponseRef.java @@ -0,0 +1,77 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.project; + +import com.google.common.collect.Maps; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractResponseRef; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class VisualisProjectResponseRef extends AbstractResponseRef implements ProjectResponseRef { + + private static final Logger LOGGER = LoggerFactory.getLogger(VisualisProjectResponseRef.class); + private Long projectRefId; + private AppInstance appInstance; + private String errorMsg; + + protected VisualisProjectResponseRef(String responseBody, int status) throws Exception { + super(responseBody, status); + responseMap = BDPJettyServerHelper.jacksonJson().readValue(responseBody, Map.class); + } + + @Override + public Long getProjectRefId() { + return projectRefId; + } + + @Override + public Map getProjectRefIds() { + Map projectRefIdsMap = Maps.newHashMap(); + projectRefIdsMap.put(appInstance, projectRefId); + return projectRefIdsMap; + } + + @Override + public Map toMap() { + return responseMap; + } + + @Override + public String getErrorMsg() { + return errorMsg; + } + + public void setProjectRefId(Long projectRefId) { + this.projectRefId = projectRefId; + } + + public AppInstance getAppInstance() { + return appInstance; + } + + public void setAppInstance(AppInstance appInstance) { + this.appInstance = appInstance; + } + + public void setErrorMsg(String errorMsg) { + this.errorMsg = errorMsg; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectService.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectService.java new file mode 100644 index 000000000..09372ed34 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/project/VisualisProjectService.java @@ -0,0 +1,63 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.project; + +import com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectUrlOperation; +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction; +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult; + +public class VisualisProjectService extends ProjectService { + + @Override + public boolean isCooperationSupported() { + return true; + } + + @Override + public boolean isProjectNameUnique() { + return false; + } + + @Override + protected VisualisProjectCreationOperation createProjectCreationOperation() { + SSORequestOperation ssoRequestOperation = getSSORequestService().createSSORequestOperation(VisualisAppConn.VISUALIS_APPCONN_NAME); + VisualisProjectCreationOperation visualisProjectCreationOperation = new VisualisProjectCreationOperation(this, ssoRequestOperation); + visualisProjectCreationOperation.setStructureService(this); + return visualisProjectCreationOperation; + } + + @Override + protected ProjectUpdateOperation createProjectUpdateOperation() { + return null; + } + + @Override + protected ProjectDeletionOperation createProjectDeletionOperation() { + return null; + } + + @Override + protected ProjectUrlOperation createProjectUrlOperation() { + return null; + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/publish/VisualisExportResponseRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/publish/VisualisExportResponseRef.java new file mode 100644 index 000000000..42814d7a7 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/publish/VisualisExportResponseRef.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.publish; + +import com.webank.wedatasphere.dss.standard.app.development.ref.DSSCommonResponseRef; + +import java.util.Map; + +public class VisualisExportResponseRef extends DSSCommonResponseRef { + + Map bmlResource; + + public VisualisExportResponseRef(String responseBody) throws Exception { + super(responseBody); + bmlResource = ((Map) responseMap.get("data")); + } + + @Override + public Map toMap() { + return bmlResource; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/publish/VisualisImportResponseRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/publish/VisualisImportResponseRef.java new file mode 100644 index 000000000..2f4a81b2c --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/publish/VisualisImportResponseRef.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.publish; + +import com.google.common.collect.Maps; +import com.webank.wedatasphere.dss.standard.app.development.ref.DSSCommonResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +import java.util.Map; + +public class VisualisImportResponseRef extends DSSCommonResponseRef { + + Map importedMap = Maps.newHashMap(); + Map newJobContent = Maps.newHashMap(); + + public VisualisImportResponseRef(Map jobContent, String responseBody, String nodeType, Object projectId) throws Exception { + super(responseBody); + + if("linkis.appconn.visualis.widget".equalsIgnoreCase(nodeType)){ + Map payload = (Map) jobContent.get("data"); + Long id = ((Double) Double.parseDouble(payload.get("widgetId").toString())).longValue(); + payload.put("widgetId", ((Double) ((Map) ((Map) responseMap.get("data")).get("widget")).get(id.toString()).doubleValue()).toString()); + } else if("linkis.appconn.visualis.display".equalsIgnoreCase(nodeType)){ + Map payload = (Map) jobContent.get("payload"); + Long id = ((Double) Double.parseDouble(payload.get("id").toString())).longValue(); + payload.put("projectId", projectId); + payload.put("id", ((Double) ((Map) ((Map) responseMap.get("data")).get("display")).get(id.toString()).doubleValue()).toString()); + } else if("linkis.appconn.visualis.dashboard".equalsIgnoreCase(nodeType)){ + Map payload = (Map) jobContent.get("payload"); + Long id = ((Double) Double.parseDouble(payload.get("id").toString())).longValue(); + payload.put("projectId", projectId); + payload.put("id", ((Double) ((Map) ((Map) responseMap.get("data")).get("dashboardPortal")).get(id.toString()).doubleValue()).toString()); + } else { + throw new ExternalOperationFailedException(90177, "Unknown task type " + nodeType, null); + } + this.newJobContent = jobContent; + } + + @Override + public Map toMap() { + return newJobContent; + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/NodeUpdateCSRequestRefImpl.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/NodeUpdateCSRequestRefImpl.java new file mode 100644 index 000000000..7aa6210f1 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/NodeUpdateCSRequestRefImpl.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.UpdateCSRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.NodeRequestRefImpl; + + +public class NodeUpdateCSRequestRefImpl extends NodeRequestRefImpl implements UpdateCSRequestRef { + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisCommonResponseRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisCommonResponseRef.java new file mode 100644 index 000000000..37afbffda --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisCommonResponseRef.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.ref; + +import com.webank.wedatasphere.dss.appconn.visualis.utils.VisualisNodeUtils; +import com.webank.wedatasphere.dss.standard.app.development.ref.DSSCommonResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public class VisualisCommonResponseRef extends DSSCommonResponseRef { + + + public VisualisCommonResponseRef(String responseBody) throws Exception { + super(responseBody); + } + + public String getWidgetId() throws ExternalOperationFailedException { + return VisualisNodeUtils.getWidgetId(responseBody); + } + + public String getDisplayId() throws ExternalOperationFailedException { + return VisualisNodeUtils.getDisplayId(responseBody); + } + + public String getDashboardId() throws ExternalOperationFailedException { + return VisualisNodeUtils.getDashboardPortalId(responseBody); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisOpenRequestRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisOpenRequestRef.java new file mode 100644 index 000000000..8de783392 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisOpenRequestRef.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; +import com.webank.wedatasphere.dss.standard.app.development.ref.OpenRequestRef; + +import java.util.Map; + +public class VisualisOpenRequestRef extends CommonRequestRefImpl implements OpenRequestRef { + + @Override + public String getName() { + return ((Map)this.getParameters().get("params")).get("title").toString(); + } + + @Override + public String getType() { + return ((Map)this.getParameters().get("node")).get("nodeType").toString(); + } + + public Object getJobContent() { + return ((Map)this.getParameters().get("params")); + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisOpenResponseRef.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisOpenResponseRef.java new file mode 100644 index 000000000..d5e94eb3c --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/ref/VisualisOpenResponseRef.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + +public class VisualisOpenResponseRef extends CommonResponseRef { + + public VisualisOpenResponseRef(String responseBody, int status) { + super(responseBody, status); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisCRUDService.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisCRUDService.java new file mode 100644 index 000000000..45a2c8c06 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisCRUDService.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.service; + +import com.webank.wedatasphere.dss.appconn.visualis.operation.VisualisRefCreationOperation; +import com.webank.wedatasphere.dss.appconn.visualis.operation.VisualisRefDeletionOperation; +import com.webank.wedatasphere.dss.appconn.visualis.operation.VisualisRefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCopyOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefCRUDService; + +public class VisualisCRUDService extends AbstractRefCRUDService { + + @Override + protected RefCreationOperation createRefCreationOperation() { + return new VisualisRefCreationOperation(this); + } + + @Override + protected RefCopyOperation createRefCopyOperation() { + return null; + } + + @Override + protected RefUpdateOperation createRefUpdateOperation() { + return new VisualisRefUpdateOperation(this); + } + + @Override + protected RefDeletionOperation createRefDeletionOperation() { + return new VisualisRefDeletionOperation(this); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisQueryService.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisQueryService.java new file mode 100644 index 000000000..2a710cc74 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisQueryService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.service; + +import com.webank.wedatasphere.dss.appconn.visualis.operation.VisualisRefQueryOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefQueryService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; + +public class VisualisQueryService extends AbstractRefQueryService { + + @Override + public RefQueryOperation createRefQueryOperation() { + return new VisualisRefQueryOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisRefExportService.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisRefExportService.java new file mode 100644 index 000000000..539047004 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisRefExportService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.service; + +import com.webank.wedatasphere.dss.appconn.visualis.operation.VisualisRefExportOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExportService; + +public class VisualisRefExportService extends AbstractRefExportService { + + @Override + public RefExportOperation createRefExportOperation() { + return new VisualisRefExportOperation(this); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisRefImportService.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisRefImportService.java new file mode 100644 index 000000000..ee2ceb839 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/service/VisualisRefImportService.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.service; + +import com.webank.wedatasphere.dss.appconn.visualis.operation.VisualisRefImportOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefImportService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; + +public class VisualisRefImportService extends AbstractRefImportService { + + @Override + protected RefImportOperation createRefImportOperation() { + VisualisRefImportOperation visualisRefImportOperation = new VisualisRefImportOperation(this); + return visualisRefImportOperation; + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/HttpUtils.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/HttpUtils.java new file mode 100644 index 000000000..11db0ae6c --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/HttpUtils.java @@ -0,0 +1,133 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.utils; + +import org.apache.commons.io.IOUtils; +import org.apache.http.HttpEntity; +import org.apache.http.client.CookieStore; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpDelete; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.client.methods.HttpPut; +import org.apache.http.entity.StringEntity; +import org.apache.http.impl.client.BasicCookieStore; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.protocol.HTTP; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; + +public class HttpUtils { + private static Logger logger = LoggerFactory.getLogger(HttpUtils.class); + public static String sendPostReq(String url, String params, + String user) throws Exception { + String resultString = "{}"; + logger.info("sendPostReq url is: "+url); + HttpPost httpPost = new HttpPost(url); + httpPost.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); + httpPost.addHeader("Token-User",user); + httpPost.addHeader("Token-Code", "WS-AUTH"); + CookieStore cookieStore = new BasicCookieStore(); + logger.info("Http request params is :"+params); + StringEntity entity = entity = new StringEntity(params); + entity.setContentEncoding("UTF-8"); + entity.setContentType("application/json"); + httpPost.setEntity(entity); + CloseableHttpClient httpClient = null; + CloseableHttpResponse response = null; + try { + httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); + response = httpClient.execute(httpPost); + HttpEntity ent = response.getEntity(); + resultString = IOUtils.toString(ent.getContent(), "utf-8"); + logger.info("Send Http Request Success", resultString); + } catch (Exception e) { + logger.error("Send Http Request Failed", e); + throw e; + } finally { + IOUtils.closeQuietly(response); + IOUtils.closeQuietly(httpClient); + } + return resultString; + } + + public static String sendHttpDelete(String url,String user) throws Exception { + String resultString = "{}"; + HttpDelete httpdelete = new HttpDelete(url); + logger.info("sendDeleteReq url is: "+url); + httpdelete.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); + httpdelete.addHeader("Token-User",user); + httpdelete.addHeader("Token-Code","WS-AUTH"); + CookieStore cookieStore = new BasicCookieStore(); + CloseableHttpClient httpClient = null; + CloseableHttpResponse response = null; + try { + httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); + response = httpClient.execute(httpdelete); + HttpEntity ent = response.getEntity(); + resultString = IOUtils.toString(ent.getContent(), "utf-8"); + logger.info("Send Http Delete Request Success", resultString); + } catch (Exception e) { + logger.error("Send Http Delete Request Failed", e); + throw e; + }finally{ + IOUtils.closeQuietly(response); + IOUtils.closeQuietly(httpClient); + } + return resultString; + } + + public static String sendHttpPut(String url, String params, + String user) throws Exception { + String resultString = "{}"; + logger.info("sendPostReq url is: "+url); + HttpPut httpPut = new HttpPut(url); + httpPut.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); + httpPut.addHeader("Token-User",user); + httpPut.addHeader("Token-Code","WS-AUTH"); + CookieStore cookieStore = new BasicCookieStore(); + logger.info("Http put params is :"+params); + StringEntity entity = null; + try { + entity = new StringEntity(params); + } catch (UnsupportedEncodingException e) { + throw e; + } + entity.setContentEncoding("UTF-8"); + entity.setContentType("application/json"); + httpPut.setEntity(entity); + CloseableHttpClient httpClient = null; + CloseableHttpResponse response = null; + try { + httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); + response = httpClient.execute(httpPut); + HttpEntity ent = response.getEntity(); + resultString = IOUtils.toString(ent.getContent(), "utf-8"); + logger.info("Send Http Put Success", resultString); + } catch (Exception e) { + logger.error("Send Http Put Failed", e); + throw e; + } finally { + IOUtils.closeQuietly(response); + IOUtils.closeQuietly(httpClient); + } + return resultString; + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/NumberUtils.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/NumberUtils.java new file mode 100644 index 000000000..2dea0fb60 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/NumberUtils.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.utils; + +public class NumberUtils { + + public static Integer getInt(Object original){ + if(original instanceof Double){ + return ((Double) original).intValue(); + } + return (Integer) original; + } + + public static String parseDoubleString(String doubleString) { + Double doubleValue = Double.parseDouble(doubleString); + Integer intValue = doubleValue.intValue(); + return intValue.toString(); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/URLUtils.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/URLUtils.java new file mode 100644 index 000000000..8ed8064f9 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/URLUtils.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.utils; + +import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration; + +public class URLUtils { + public final static String widgetUrl = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widget" + "/smartcreate"; + public final static String widgetUpdateUrl = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widget" + "/rename"; + public final static String widgetContextUrl = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widget" + "/setcontext"; + public final static String widgetDeleteUrl = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widgets"; + public final static String displayUrl = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/displays"; + public final static String dashboardPortalUrl = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/dashboardPortals"; + public final static String displaySlideConfig = "{\"slideParams\":{\"width\":1920,\"height\":1080,\"backgroundColor\":[255,255,255],\"scaleMode\":\"noScale\",\"backgroundImage\":null}}"; + public final static String projectUrl = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/project"; + + public final static String DISPLAY_PREVIEW_URL_FORMAT = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/displays/%s/preview"; + public final static String DASHBOARD_PREVIEW_URL_FORMAT = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/dashboard/portal/%s/preview"; + public final static String WIDGET_DATA_URL_FORMAT = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widget/%s/getdata"; + public final static String DISPLAY_METADATA_URL_FORMAT = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widget/display/%s/metadata"; + public final static String DASHBOARD_METADATA_URL_FORMAT = "/api/rest_j/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/widget/portal/%s/metadata"; + + public final static String WIDGET_JUMP_URL_FORMAT = "dss/visualis/#/project/%s/widget/%s"; + public final static String DISPLAY_JUMP_URL_FORMAT = "dss/visualis/#/project/%s/display/%s"; + public final static String DASHBOARD_JUMP_URL_FORMAT = "dss/visualis/#/project/%s/portal/%s/portalName/%s"; + + + public static String getUrl(String baseUrl, String format, String entityId){ + return baseUrl + String.format(format, entityId); + } + + public static String getUrl(String baseUrl, String format, String... ids){ + return baseUrl + String.format(format, ids); + } + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/VisualisDownloadAction.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/VisualisDownloadAction.java new file mode 100644 index 000000000..826957eb3 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/VisualisDownloadAction.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.utils; + +import com.webank.wedatasphere.linkis.httpclient.request.DownloadAction; +import com.webank.wedatasphere.linkis.httpclient.request.GetAction; +import com.webank.wedatasphere.linkis.httpclient.request.UserAction; + +import java.io.InputStream; + +public class VisualisDownloadAction extends GetAction implements DownloadAction, UserAction { + + private String user; + private String url; + private InputStream inputStream; + + @Override + public void write(InputStream inputStream) { + this.inputStream = inputStream; + } + + public InputStream getInputStream() { + return inputStream; + } + + @Override + public String getUser() { + return user; + } + + @Override + public void setUser(String user) { + this.user = user; + } + + public void setURL(String url) { + this.url = url; + } + + @Override + public String getURL() { + return url; + } +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/VisualisNodeUtils.java b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/VisualisNodeUtils.java new file mode 100644 index 000000000..4eca9e9dc --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/visualis/utils/VisualisNodeUtils.java @@ -0,0 +1,84 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.visualis.utils; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonMappingException; +import com.webank.wedatasphere.dss.standard.app.development.ref.NodeRequestRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; + +import java.util.Map; + +public class VisualisNodeUtils { + + public static String getId(NodeRequestRef nodeRequestRef) throws Exception { + String externalContent = BDPJettyServerHelper.jacksonJson().writeValueAsString(nodeRequestRef.getJobContent()); + if ("linkis.appconn.visualis.display".equalsIgnoreCase(nodeRequestRef.getNodeType())) { + return NumberUtils.parseDoubleString(getDisplayId(externalContent)); + } else if ("linkis.appconn.visualis.dashboard".equalsIgnoreCase(nodeRequestRef.getNodeType())) { + return NumberUtils.parseDoubleString(getDashboardPortalId(externalContent)); + } else if ("linkis.appconn.visualis.widget".equalsIgnoreCase(nodeRequestRef.getNodeType())) { + return NumberUtils.parseDoubleString(getWidgetId(externalContent)); + } + return null; + } + + + public static String getDisplayId(String responseBody) throws ExternalOperationFailedException { + String displayId = null; + try { + Map responseMap = BDPJettyServerHelper.jacksonJson().readValue(responseBody, Map.class); + displayId = ((Map) responseMap.get("payload")).get("id").toString(); + } catch (JsonMappingException e) { + throw new ExternalOperationFailedException(1000054, "Get Display Id failed!", e); + } catch (JsonProcessingException e) { + throw new ExternalOperationFailedException(1000054, "Get Display Id failed!", e); + } + + return displayId; + } + + public static String getWidgetId(String responseBody) throws ExternalOperationFailedException { + String widgetId = null; + try { + Map responseMap = BDPJettyServerHelper.jacksonJson().readValue(responseBody, Map.class); + widgetId = ((Map) responseMap.get("data")).get("widgetId").toString(); + } catch (JsonMappingException e) { + throw new ExternalOperationFailedException(1000055, "Get widget Id failed!", e); + } catch (JsonProcessingException e) { + throw new ExternalOperationFailedException(1000055, "Get widget Id failed!", e); + } + return widgetId; + } + + public static String getDashboardPortalId(String responseBody) throws ExternalOperationFailedException { + String dashboardPortalId = null; + try { + Map responseMap = BDPJettyServerHelper.jacksonJson().readValue(responseBody, Map.class); + dashboardPortalId = ((Map) responseMap.get("payload")).get("id").toString(); + } catch (JsonMappingException e) { + throw new ExternalOperationFailedException(1000056, "Get dashboard Id failed!", e); + } catch (JsonProcessingException e) { + throw new ExternalOperationFailedException(1000056, "Get dashboard Id failed!", e); + } + + return Long.toString(Math.round(Double.parseDouble(dashboardPortalId))); + } + + +} diff --git a/dss-appconn/appconns/dss-visualis-appconn/src/main/resources/init.sql b/dss-appconn/appconns/dss-visualis-appconn/src/main/resources/init.sql new file mode 100644 index 000000000..667e818a2 --- /dev/null +++ b/dss-appconn/appconns/dss-visualis-appconn/src/main/resources/init.sql @@ -0,0 +1,78 @@ + +delete from `dss_application` WHERE `name` ='visualis'; +INSERT INTO `dss_application`(`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES ('visualis','',0,1,NULL,0,'','{\"watermark\":false,\"rsDownload\":true}',1,'',NULL); + +UPDATE `dss_application` SET url = 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT' WHERE `name` ='visualis'; +UPDATE `dss_application` SET project_url = 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/dss/visualis/#/project/${projectId}',homepage_url = 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/dss/visualis/#/projects' WHERE `name` in('visualis'); + +select @dss_visualis_applicationId:=id from `dss_application` WHERE `name` ='visualis'; + +delete from `dss_menu` WHERE `name` ='visualis'; +INSERT INTO `dss_menu`(`name`,`level`,`upper_menu_id`,`front_name`,`comment`,`description`,`is_active`,`is_component`,`icon`,`application_id`) values ('visualis','2',3,'知画(Visualis)',NULL,NULL,1,1,NULL,@dss_visualis_applicationId); + +delete from `dss_onestop_menu_application` WHERE title_en='Visualis'; +INSERT INTO `dss_onestop_menu_application` (`application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) VALUES(@dss_visualis_applicationId,'2','Visualis','Visualis','Visualis is a data visualization BI tool based on Davinci, with Linkis as the kernel, it supports the analysis mode of data development exploration.','Visualis是基于宜信开源项目Davinci开发的数据可视化BI工具，以任意桥(Linkis)做为内核，支持拖拽式报表定义、图表联动、钻取、全局筛选、多维分析、实时查询等数据开发探索的分析模式，并做了水印、数据质量校验等金融级增强。','visualization, statement','可视化,报表','1','enter Visualis','进入Visualis','user manual','用户手册','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujukeshihua-logo',NULL,NULL,NULL,NULL,NULL,'shujukeshihua-icon'); + + +delete from `dss_appconn` where `appconn_name`='visualis'; +INSERT INTO `dss_appconn` (`appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, `reference`, `class_name`, `appconn_class_path`, `resource`) VALUES ('visualis', 0, 1, NULL, 0, NULL, 'com.webank.wedatasphere.dss.appconn.visualis.VisualisAppConn', 'DSS_INSTALL_HOME_VAL/dss-appconns/visualis/lib', ''); + +select @dss_appconn_visualisId:=id from `dss_appconn` where `appconn_name` = 'visualis'; + +delete from `dss_appconn_instance` where `homepage_url` like '%visualis%'; +INSERT INTO `dss_appconn_instance` (`appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url`) VALUES (@dss_appconn_visualisId, 'DEV', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/', '', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/dss/visualis/#/projects', 'http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/'); + + +delete from `dss_workflow_node` where `node_type` like '%visualis%'; +insert into `dss_workflow_node` (`name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('display','visualis','linkis.appconn.visualis.display','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/dss/visualis/#/project/${projectId}/display/${displayId}','1','1','0','1','displayCreated with Sketch.'); +insert into `dss_workflow_node` (`name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('dashboard','visualis','linkis.appconn.visualis.dashboard','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/dss/visualis/#/project/${projectId}/portal/${dashboardPortalId}/portalName/${nodeName}','1','1','0','1','dashboardCreated with Sketch.'); +insert into `dss_workflow_node` (`name`, `appconn_name`, `node_type`, `jump_url`, `support_jump`, `submit_to_scheduler`, `enable_copy`, `should_creation_before_node`, `icon`) values('widget','visualis','linkis.appconn.visualis.widget','http://APPCONN_INSTALL_IP:APPCONN_INSTALL_PORT/dss/visualis/#/project/${projectId}/widget/${widgetId}','1','1','0','1','widgetCreated with Sketch.'); + + + +select @dss_visualis_displayId:=id from `dss_workflow_node` where `node_type` = 'linkis.appconn.visualis.display'; +select @dss_visualis_dashboardId:=id from `dss_workflow_node` where `node_type` = 'linkis.appconn.visualis.dashboard'; +select @dss_visualis_dashwidgetId:=id from `dss_workflow_node` where `node_type` = 'linkis.appconn.visualis.widget'; + +delete from `dss_workflow_node_to_group` where `node_id`=@dss_visualis_displayId; +delete from `dss_workflow_node_to_group` where `node_id`=@dss_visualis_dashboardId; +delete from `dss_workflow_node_to_group` where `node_id`=@dss_visualis_dashwidgetId; + + +delete from `dss_workflow_node_to_ui` where `workflow_node_id`=@dss_visualis_displayId; +delete from `dss_workflow_node_to_ui` where `workflow_node_id`=@dss_visualis_dashboardId; +delete from `dss_workflow_node_to_ui` where `workflow_node_id`=@dss_visualis_dashwidgetId; + + +select @dss_visualis_displayId:=id from `dss_workflow_node` where `node_type` = 'linkis.appconn.visualis.display'; +select @dss_visualis_dashboardId:=id from `dss_workflow_node` where `node_type` = 'linkis.appconn.visualis.dashboard'; +select @dss_visualis_dashwidgetId:=id from `dss_workflow_node` where `node_type` = 'linkis.appconn.visualis.widget'; + +INSERT INTO `dss_workflow_node_to_group`(`node_id`,`group_id`) values (@dss_visualis_displayId,4); +INSERT INTO `dss_workflow_node_to_group`(`node_id`,`group_id`) values (@dss_visualis_dashboardId,4); +INSERT INTO `dss_workflow_node_to_group`(`node_id`,`group_id`) values (@dss_visualis_dashwidgetId,4); + + +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,1); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,2); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,3); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,4); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,5); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,6); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_displayId,45); + +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashboardId,1); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashboardId,2); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashboardId,3); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashboardId,4); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashboardId,5); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashboardId,45); + + +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,1); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,3); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,5); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,6); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,37); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,2); +INSERT INTO `dss_workflow_node_to_ui`(`workflow_node_id`,`ui_id`) values (@dss_visualis_dashwidgetId,45); diff --git a/dss-appconn/appconns/dss-workflow-appconn/pom.xml b/dss-appconn/appconns/dss-workflow-appconn/pom.xml new file mode 100644 index 000000000..5cf95329b --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/pom.xml @@ -0,0 +1,112 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-workflow-appconn + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + provided + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-sender-service + ${dss.version} + provided + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + \ No newline at end of file diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/assembly/distribution.xml b/dss-appconn/appconns/dss-workflow-appconn/src/main/assembly/distribution.xml new file mode 100644 index 000000000..003e51313 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/assembly/distribution.xml @@ -0,0 +1,66 @@ + + + + dss-workflow-appconn + + dir + + true + workflow + + + + + + lib + true + true + false + true + true + + + + + + ${basedir}/src/main/resources + + appconn.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + log4j.properties + log4j2.xml + + 0777 + conf + unix + + + + + + diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/DefaultWorkflowAppConn.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/DefaultWorkflowAppConn.java new file mode 100644 index 000000000..8198af3b9 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/DefaultWorkflowAppConn.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow; + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractAppConn; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppDescImpl; + +public class DefaultWorkflowAppConn extends AbstractAppConn implements OnlyDevelopmentAppConn { + + @Override + protected void initialize() { + AppDescImpl appDesc = new AppDescImpl(); + appDesc.setAppName("WorkflowAppConn"); + setAppDesc(appDesc); + } + + @Override + public DevelopmentIntegrationStandard getOrCreateDevelopmentStandard() { + return new WorkflowDevelopmentIntegrationStandard(); + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/WorkflowDevelopmentIntegrationStandard.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/WorkflowDevelopmentIntegrationStandard.java new file mode 100644 index 000000000..f93d4c396 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/WorkflowDevelopmentIntegrationStandard.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow; + +import com.webank.wedatasphere.dss.appconn.workflow.service.WorkflowCRUDService; +import com.webank.wedatasphere.dss.appconn.workflow.service.WorkflowExportService; +import com.webank.wedatasphere.dss.appconn.workflow.service.WorkflowImportService; +import com.webank.wedatasphere.dss.appconn.workflow.service.WorkflowQueryService; +import com.webank.wedatasphere.dss.standard.app.development.service.*; +import com.webank.wedatasphere.dss.standard.app.development.standard.AbstractDevelopmentIntegrationStandard; + +public class WorkflowDevelopmentIntegrationStandard extends AbstractDevelopmentIntegrationStandard { + + @Override + protected RefCRUDService createRefCRUDService() { + return new WorkflowCRUDService(); + } + + @Override + protected RefExecutionService createRefExecutionService() { + return null; + } + + @Override + protected RefExportService createRefExportService() { + return new WorkflowExportService(); + } + + @Override + protected RefImportService createRefImportService() { + return new WorkflowImportService(); + } + + @Override + protected RefQueryService createRefQueryService() { + return new WorkflowQueryService(); + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefExportOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefExportOperation.java new file mode 100644 index 000000000..23ce611d7 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefExportOperation.java @@ -0,0 +1,80 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowExportRequestRef; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowExportResponseRef; +import com.webank.wedatasphere.dss.common.protocol.RequestExportWorkflow; +import com.webank.wedatasphere.dss.common.protocol.ResponseExportWorkflow; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class WorkflowRefExportOperation implements RefExportOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(WorkflowRefExportOperation.class); + + private DevelopmentService developmentService; + + + @Override + public WorkflowExportResponseRef exportRef(WorkflowExportRequestRef workflowExportRequestRef) throws ExternalOperationFailedException { + + String userName = workflowExportRequestRef.getUserName(); + //todo + long flowId = workflowExportRequestRef.getAppId(); + Long projectId = workflowExportRequestRef.getProjectId(); + String projectName = workflowExportRequestRef.getProjectName(); + RequestExportWorkflow requestExportWorkflow = new RequestExportWorkflow(userName, + flowId, + projectId, + projectName, + BDPJettyServerHelper.gson().toJson(workflowExportRequestRef.getWorkspace()), + workflowExportRequestRef.getDSSLabels()); + ResponseExportWorkflow responseExportWorkflow = null; + try{ + Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getWorkflowSender(workflowExportRequestRef.getDSSLabels()); + responseExportWorkflow = (ResponseExportWorkflow) sender.ask(requestExportWorkflow); + }catch(final Exception t){ + DSSExceptionUtils.dealErrorException(60025, "failed to get rpc message", t, ExternalOperationFailedException.class); + } + if (null != responseExportWorkflow) { + WorkflowExportResponseRef workflowExportResponseRef = new WorkflowExportResponseRef(); + workflowExportResponseRef.setFlowID(responseExportWorkflow.flowID()); + workflowExportResponseRef.setResourceId(responseExportWorkflow.resourceId()); + workflowExportResponseRef.setVersion(responseExportWorkflow.version()); + workflowExportResponseRef.addResponse("resourceId", responseExportWorkflow.resourceId()); + workflowExportResponseRef.addResponse("version", responseExportWorkflow.version()); + workflowExportResponseRef.addResponse("flowID", responseExportWorkflow.flowID()); + return workflowExportResponseRef; + } else { + throw new ExternalOperationFailedException(100085, "Error ask workflow to export!", null); + } + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefImportOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefImportOperation.java new file mode 100644 index 000000000..f26266e1b --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefImportOperation.java @@ -0,0 +1,71 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowImportResponseRef; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowImportRequestRef; +import com.webank.wedatasphere.dss.workflow.common.protocol.RequestImportWorkflow; +import com.webank.wedatasphere.dss.workflow.common.protocol.ResponseImportWorkflow; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class WorkflowRefImportOperation implements + RefImportOperation { + + private static final Logger LOGGER = LoggerFactory.getLogger(WorkflowRefImportOperation.class); + + private DevelopmentService developmentService; + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + @Override + public WorkflowImportResponseRef importRef(WorkflowImportRequestRef requestRef) throws ExternalOperationFailedException { + + WorkflowImportResponseRef workflowImportResponseRef = null; + RequestImportWorkflow requestImportWorkflow = new RequestImportWorkflow(requestRef.getUserName(), + requestRef.getResourceId(), requestRef.getBmlVersion(), + requestRef.getProjectId(), requestRef.getProjectName(), + requestRef.getSourceEnv(), requestRef.getOrcVersion(), + requestRef.getWorkspaceName(), + DSSCommonUtils.COMMON_GSON.toJson(requestRef.getWorkspace()), + requestRef.getContextID()); + + Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getSchedulerWorkflowSender(); + if (null != sender) { + ResponseImportWorkflow responseImportWorkflow = (ResponseImportWorkflow) sender.ask(requestImportWorkflow); + workflowImportResponseRef = new WorkflowImportResponseRef(); + if (responseImportWorkflow.getWorkflowIds() != null && responseImportWorkflow.getWorkflowIds().size() > 0){ + workflowImportResponseRef.setOrcId(responseImportWorkflow.getWorkflowIds().get(0)); + }else{ + LOGGER.error("failed to get ref orc id, workflow Ids are {}", responseImportWorkflow.getWorkflowIds()); + } + workflowImportResponseRef.setStatus(responseImportWorkflow.getStatus()); + } else { + throw new ExternalOperationFailedException(100039, "Rpc sender is null", null); + } + return workflowImportResponseRef; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefQueryOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefQueryOperation.java new file mode 100644 index 000000000..b5fb79ac6 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowRefQueryOperation.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowUrlResponseRef; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.OpenRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class WorkflowRefQueryOperation implements RefQueryOperation { + + protected final Logger LOGGER = LoggerFactory.getLogger(getClass()); + + private DevelopmentService developmentService; + + @Override + public WorkflowUrlResponseRef query(OpenRequestRef ref) throws ExternalOperationFailedException { + EnvDSSLabel label = (EnvDSSLabel) ref.getDSSLabels().stream().filter(EnvDSSLabel.class::isInstance) + .findFirst().orElseThrow(() -> new DSSRuntimeException(50321, "Not exists EnvDSSLabel.")); + String urlStr = "router/workflow/editable?labels=" + label.getEnv(); + if (LOGGER.isDebugEnabled()){ + LOGGER.debug("url for {} is {}", ref.getName(), urlStr); + } + WorkflowUrlResponseRef workflowUrlResponseRef = new WorkflowUrlResponseRef(); + workflowUrlResponseRef.setUrl(urlStr); + return workflowUrlResponseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService =service; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskCopyOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskCopyOperation.java new file mode 100644 index 000000000..7d1839ce4 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskCopyOperation.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowCopyRequestRef; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowCopyResponseRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCopyOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.workflow.common.protocol.RequestCopyWorkflow; +import com.webank.wedatasphere.dss.workflow.common.protocol.ResponseCopyWorkflow; +import com.webank.wedatasphere.linkis.rpc.Sender; + + +public class WorkflowTaskCopyOperation implements RefCopyOperation { + private DevelopmentService service; + + private final Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getWorkflowSender(); + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.service = service; + } + + @Override + public WorkflowCopyResponseRef copyRef(WorkflowCopyRequestRef workflowCopyRequestRef) { + Long appId = workflowCopyRequestRef.getAppId(); + Long orcVersionId = workflowCopyRequestRef.getOrcVersionId(); + String userName = workflowCopyRequestRef.getUserName(); + String workspaceName = workflowCopyRequestRef.getWorkspaceName(); + String contextIdStr = workflowCopyRequestRef.getContextID(); + String projectName = workflowCopyRequestRef.getProjectName(); + String version = workflowCopyRequestRef.getVersion(); + String description = workflowCopyRequestRef.getDescription(); + RequestCopyWorkflow requestCopyWorkflow = new RequestCopyWorkflow(userName, + workspaceName, appId, contextIdStr, + projectName, orcVersionId, + version, description); + ResponseCopyWorkflow responseCopyWorkflow = (ResponseCopyWorkflow) sender.ask(requestCopyWorkflow); + WorkflowCopyResponseRef workflowCopyResponseRef = new WorkflowCopyResponseRef(); + workflowCopyResponseRef.setDssFlow(responseCopyWorkflow.getDssFlow()); + workflowCopyResponseRef.setCopyTargetAppId(responseCopyWorkflow.getDssFlow().getId()); + return workflowCopyResponseRef; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskCreationOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskCreationOperation.java new file mode 100644 index 000000000..eb649a119 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskCreationOperation.java @@ -0,0 +1,77 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.ref.DefaultOrchestratorCreateRequestRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.workflow.common.protocol.RequestCreateWorkflow; +import com.webank.wedatasphere.dss.workflow.common.protocol.ResponseCreateWorkflow; +import com.webank.wedatasphere.linkis.rpc.Sender; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; + +public class WorkflowTaskCreationOperation implements RefCreationOperation { + private static final Logger LOGGER = LoggerFactory.getLogger(WorkflowTaskCreationOperation.class); + + private Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getWorkflowSender(); + + private DevelopmentService developmentService; + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } + + @Override + public CommonResponseRef createRef(DefaultOrchestratorCreateRequestRef workflowCreateRequestRef) throws ExternalOperationFailedException { + //发送RPC请求 + CommonResponseRef workflowCreateResponseRef = null; + String userName = workflowCreateRequestRef.getUserName(); + String workflowName = workflowCreateRequestRef.getDSSOrchestratorInfo().getName(); + String contextIDStr = workflowCreateRequestRef.getContextID() != null ? + workflowCreateRequestRef.getContextID() : ""; + String description = workflowCreateRequestRef.getDSSOrchestratorInfo().getDesc(); + List dssLabels = workflowCreateRequestRef.getDSSLabels(); + long parentFlowId = -1L; + List linkedAppConnNames = workflowCreateRequestRef.getDSSOrchestratorInfo().getLinkedAppConnNames() != null ? + workflowCreateRequestRef.getDSSOrchestratorInfo().getLinkedAppConnNames() : new ArrayList<>(); + String uses = workflowCreateRequestRef.getDSSOrchestratorInfo().getUses() != null ? + workflowCreateRequestRef.getDSSOrchestratorInfo().getUses() : "uses"; + RequestCreateWorkflow requestCreateWorkflow = new RequestCreateWorkflow(userName, workflowName, + contextIDStr, description, parentFlowId, uses, linkedAppConnNames, dssLabels); + + if (null != sender) { + ResponseCreateWorkflow responseCreateWorkflow = (ResponseCreateWorkflow) sender.ask(requestCreateWorkflow); + workflowCreateResponseRef = new CommonResponseRef(); + workflowCreateResponseRef.setOrcId(responseCreateWorkflow.getDssFlow().getId()); + workflowCreateResponseRef.setContent(responseCreateWorkflow.getDssFlow().getFlowJson()); + } else { + LOGGER.error("dss workflow server dev sender is null, can not send rpc message"); + DSSExceptionUtils.dealErrorException(61123,"dss workflow server dev sender is null", ExternalOperationFailedException.class); + } + return workflowCreateResponseRef; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskDeletionOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskDeletionOperation.java new file mode 100644 index 000000000..79e958e44 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskDeletionOperation.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowDeleteRequestRef; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.protocol.RequestDeleteWorkflow; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.util.List; + + +public class WorkflowTaskDeletionOperation implements RefDeletionOperation { + + private DevelopmentService developmentService; + + @Override + public void deleteRef(WorkflowDeleteRequestRef workflowDeleteRequestRef) throws ExternalOperationFailedException { + String userName = workflowDeleteRequestRef.getUserName(); + Long flowId = workflowDeleteRequestRef.getAppId(); + RequestDeleteWorkflow requestDeleteWorkflow = new RequestDeleteWorkflow(userName, flowId); + List dssLabels = workflowDeleteRequestRef.getDSSLabels(); + Sender tempSend = DSSSenderServiceFactory.getOrCreateServiceInstance().getWorkflowSender(dssLabels); + if (null != tempSend) { + tempSend.ask(requestDeleteWorkflow); + } else { + throw new ExternalOperationFailedException(100036, "Rpc sender is null", null); + } + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskUpdateOperation.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskUpdateOperation.java new file mode 100644 index 000000000..09036969f --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/opertion/WorkflowTaskUpdateOperation.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.opertion; + +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowUpdateRequestRef; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowUpdateResponseRef; +import com.webank.wedatasphere.dss.common.protocol.RequestUpdateWorkflow; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.workflow.common.protocol.ResponseUpdateWorkflow; +import com.webank.wedatasphere.linkis.rpc.Sender; + + +public class WorkflowTaskUpdateOperation implements RefUpdateOperation { + + private DevelopmentService developmentService; + + private final Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getWorkflowSender(); + + @Override + public WorkflowUpdateResponseRef updateRef(WorkflowUpdateRequestRef workflowUpdateRequestRef) throws ExternalOperationFailedException { + WorkflowUpdateResponseRef workflowUpdateResponseRef = null; + String userName = workflowUpdateRequestRef.getUserName(); + Long flowID = workflowUpdateRequestRef.getOrcId(); + String flowName = workflowUpdateRequestRef.getOrcName(); + String description = workflowUpdateRequestRef.getDescription(); + String uses = workflowUpdateRequestRef.getUses(); + RequestUpdateWorkflow requestUpdateWorkflow = new RequestUpdateWorkflow(userName, flowID, flowName, description, uses); + if (null != sender) { + ResponseUpdateWorkflow responseUpdateWorkflow = (ResponseUpdateWorkflow) sender.ask(requestUpdateWorkflow); + workflowUpdateResponseRef = new WorkflowUpdateResponseRef(); + workflowUpdateResponseRef.setJobStatus(responseUpdateWorkflow.getJobStatus()); + } else { + throw new ExternalOperationFailedException(100038, "Rpc sender is null", null); + } + return workflowUpdateResponseRef; + } + + @Override + public void setDevelopmentService(DevelopmentService service) { + this.developmentService = service; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowCopyRequestRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowCopyRequestRef.java new file mode 100644 index 000000000..ade4f9e1b --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowCopyRequestRef.java @@ -0,0 +1,76 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCopyRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class WorkflowCopyRequestRef extends CommonRequestRefImpl implements OrchestratorCopyRequestRef { + + private Long appId; + private Long orcVersionId; + private String version; + private String description; + + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Long getAppId(){ + return this.appId; + } + + + @Override + public boolean equals(Object ref) { + return false; + } + + @Override + public String toString() { + return null; + } + + @Override + public void setCopyOrcAppId(long appId) { + this.appId = appId; + } + + @Override + public void setCopyOrcVersionId(long orcVersionId) { + this.orcVersionId = orcVersionId; + } + + + public Long getOrcVersionId(){ + return orcVersionId; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowCopyResponseRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowCopyResponseRef.java new file mode 100644 index 000000000..94c4db261 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowCopyResponseRef.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCopyResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; + + +public class WorkflowCopyResponseRef extends CommonResponseRef implements OrchestratorCopyResponseRef{ + + private DSSFlow dssFlow; + + private Long copyOrcId; + + private Long copyTargetAppId; + + private String copyTargetContent; + + public DSSFlow getDssFlow() { + return dssFlow; + } + + public void setDssFlow(DSSFlow dssFlow) { + this.dssFlow = dssFlow; + } + + @Override + public long getCopyOrcId() { + return copyOrcId; + } + + public void setCopyOrcId(Long copyOrcId) { + this.copyOrcId = copyOrcId; + } + + @Override + public long getCopyTargetAppId() { + return copyTargetAppId; + } + + public void setCopyTargetAppId(Long copyTargetAppId) { + this.copyTargetAppId = copyTargetAppId; + } + + @Override + public String getCopyTargetContent() { + return copyTargetContent; + } + + public void setCopyTargetContent(String copyTargetContent) { + this.copyTargetContent = copyTargetContent; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowDeleteRequestRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowDeleteRequestRef.java new file mode 100644 index 000000000..cab8d5add --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowDeleteRequestRef.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorDeleteRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class WorkflowDeleteRequestRef extends CommonRequestRefImpl implements OrchestratorDeleteRequestRef { + + private Long appId; + + @Override + public void setAppId(Long appId) { + this.appId = appId; + } + + @Override + public Long getAppId(){ + return this.appId; + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowExportRequestRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowExportRequestRef.java new file mode 100644 index 000000000..aa1be4a12 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowExportRequestRef.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class WorkflowExportRequestRef extends CommonRequestRefImpl implements OrchestratorExportRequestRef { + private Long appId; + private Long orchestratorVersionId; + + @Override + public void setAppId(Long appId) { + this.appId =appId; + } + + @Override + public Long getAppId() { + return appId; + } + + + @Override + public void setOrchestratorVersionId(Long orchestratorVersionId) { + this.orchestratorVersionId = orchestratorVersionId; + } + + @Override + public Long getOrchestratorVersionId() { + return this.orchestratorVersionId; + } + + @Override + public boolean getAddOrcVersionFlag() { + return false; + } + + @Override + public void setAddOrcVersionFlag(boolean addOrcVersion) { + + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowExportResponseRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowExportResponseRef.java new file mode 100644 index 000000000..e540c0384 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowExportResponseRef.java @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + + +public class WorkflowExportResponseRef extends CommonResponseRef { + + private String resourceId; + + private String version; + + private Long flowID; + + public String getResourceId() { + return resourceId; + } + + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public Long getFlowID() { + return flowID; + } + + public void setFlowID(Long flowID) { + this.flowID = flowID; + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowImportRequestRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowImportRequestRef.java new file mode 100644 index 000000000..ac49f17bf --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowImportRequestRef.java @@ -0,0 +1,69 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorImportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class WorkflowImportRequestRef extends CommonRequestRefImpl implements OrchestratorImportRequestRef { + + private String resourceId; + private String bmlVersion; + private IOEnv sourceEnv; + private String orcVersion; + + @Override + public String getResourceId() { + return resourceId; + } + + @Override + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + @Override + public String getBmlVersion() { + return bmlVersion; + } + + @Override + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + @Override + public IOEnv getSourceEnv() { + return sourceEnv; + } + + @Override + public void setSourceEnv(IOEnv sourceEnv) { + this.sourceEnv = sourceEnv; + } + + @Override + public String getOrcVersion() { + return orcVersion; + } + + @Override + public void setOrcVersion(String orcVersion) { + this.orcVersion = orcVersion; + } +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowImportResponseRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowImportResponseRef.java new file mode 100644 index 000000000..06d140965 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowImportResponseRef.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + +public class WorkflowImportResponseRef extends CommonResponseRef { + + private JobStatus status; + + public void setStatus(JobStatus status) { + this.status = status; + } + + @Override + public int getStatus() { + return this.status.ordinal(); + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUpdateRequestRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUpdateRequestRef.java new file mode 100644 index 000000000..33ebd2fe7 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUpdateRequestRef.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorUpdateRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class WorkflowUpdateRequestRef extends CommonRequestRefImpl implements OrchestratorUpdateRef { + + private String description; + private String uses; + + @Override + public String getDescription() { + return description; + } + + @Override + public void setDescription(String description) { + this.description = description; + } + + @Override + public String getUses() { + return uses; + } + + @Override + public void setUses(String uses) { + this.uses = uses; + } + + @Override + public DSSOrchestratorInfo getOrchestratorInfo() { + return null; + } + + @Override + public void setOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + + } + + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUpdateResponseRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUpdateResponseRef.java new file mode 100644 index 000000000..0280faa9f --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUpdateResponseRef.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + + +public class WorkflowUpdateResponseRef extends CommonResponseRef { + private JobStatus jobStatus; + + public JobStatus getJobStatus() { + return jobStatus; + } + + public void setJobStatus(JobStatus jobStatus) { + this.jobStatus = jobStatus; + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUrlResponseRef.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUrlResponseRef.java new file mode 100644 index 000000000..c0ab9f608 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/ref/WorkflowUrlResponseRef.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.UrlResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; + +public class WorkflowUrlResponseRef extends CommonResponseRef implements UrlResponseRef { + + private String url; + + @Override + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowCRUDService.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowCRUDService.java new file mode 100644 index 000000000..56e94ef6c --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowCRUDService.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.service; + +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowTaskCopyOperation; +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowTaskDeletionOperation; +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowTaskUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCopyOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefCRUDService; +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowTaskCreationOperation; + + +public class WorkflowCRUDService extends AbstractRefCRUDService { + + @Override + protected RefCreationOperation createRefCreationOperation() { + return new WorkflowTaskCreationOperation(); + } + + @Override + protected RefCopyOperation createRefCopyOperation() { + return new WorkflowTaskCopyOperation(); + } + + @Override + protected RefUpdateOperation createRefUpdateOperation() { + return new WorkflowTaskUpdateOperation(); + } + + @Override + protected RefDeletionOperation createRefDeletionOperation() { + return new WorkflowTaskDeletionOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowExportService.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowExportService.java new file mode 100644 index 000000000..29366970f --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowExportService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.service; + +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowRefExportOperation; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefExportService; + +public class WorkflowExportService extends AbstractRefExportService { + + @Override + @SuppressWarnings("unchecked") + protected RefExportOperation createRefExportOperation() { + return new WorkflowRefExportOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowImportService.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowImportService.java new file mode 100644 index 000000000..0c6695892 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowImportService.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.service; + +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowRefImportOperation; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowImportRequestRef; +import com.webank.wedatasphere.dss.appconn.workflow.ref.WorkflowImportResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefImportService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; + +public class WorkflowImportService extends AbstractRefImportService { + + + @Override + @SuppressWarnings("unchecked") + protected RefImportOperation createRefImportOperation() { + return new WorkflowRefImportOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowQueryService.java b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowQueryService.java new file mode 100644 index 000000000..cec08b3e9 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/workflow/service/WorkflowQueryService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.workflow.service; + +import com.webank.wedatasphere.dss.appconn.workflow.opertion.WorkflowRefQueryOperation; +import com.webank.wedatasphere.dss.standard.app.development.service.AbstractRefQueryService; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; + +public class WorkflowQueryService extends AbstractRefQueryService { + + @Override + protected RefQueryOperation createRefQueryOperation() { + return new WorkflowRefQueryOperation(); + } + +} diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/appconn.properties b/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/appconn.properties new file mode 100644 index 000000000..19365e2b5 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/appconn.properties @@ -0,0 +1,20 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + + + + + diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/log4j.properties b/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/log4j.properties new file mode 100644 index 000000000..ee8619595 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO diff --git a/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/log4j2.xml b/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/log4j2.xml new file mode 100644 index 000000000..8c40a73e8 --- /dev/null +++ b/dss-appconn/appconns/dss-workflow-appconn/src/main/resources/log4j2.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-appconn/dss-appconn-core/pom.xml b/dss-appconn/dss-appconn-core/pom.xml new file mode 100644 index 000000000..e54fc31a6 --- /dev/null +++ b/dss-appconn/dss-appconn-core/pom.xml @@ -0,0 +1,106 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-appconn-core + + + 2.11.0 + 3.2.2 + + + + + com.webank.wedatasphere.dss + dss-standard-common + ${dss.version} + + + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-structure-integration-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + + commons-collections + commons-collections + ${commons-collections.version} + + + com.fasterxml.jackson.core + jackson-databind + ${fasterxml.jackson.version} + provided + + + com.fasterxml.jackson.core + jackson-annotations + ${fasterxml.jackson.version} + provided + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-origin-sso-integration-standard + ${dss.version} + + + + + + \ No newline at end of file diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/AppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/AppConn.java new file mode 100644 index 000000000..add899c08 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/AppConn.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core; + +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnErrorException; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppDesc; + +import java.util.List; + +public interface AppConn { + + void init() throws AppConnErrorException; + + /** + * 1. Get the dssappconnbean table record + * 2. Do a traversal to get all appinstances under each appconn + * 3. Instantiate the real appconn interface + */ + List getAppStandards(); + + AppDesc getAppDesc(); + + void setAppDesc(AppDesc appDesc); + +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/exception/AppConnErrorException.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/exception/AppConnErrorException.java new file mode 100644 index 000000000..129260c77 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/exception/AppConnErrorException.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.exception; + + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + +public class AppConnErrorException extends ErrorException { + + public AppConnErrorException(int errorCode, String errorDesc){ + super(errorCode, errorDesc); + } + public AppConnErrorException(int errorCode, String errorDesc, Throwable cause){ + super(errorCode, errorDesc); + initCause(cause); + } + +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/exception/AppConnWarnException.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/exception/AppConnWarnException.java new file mode 100644 index 000000000..2545d0ada --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/exception/AppConnWarnException.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.exception; + +import com.webank.wedatasphere.linkis.common.exception.WarnException; + +public class AppConnWarnException extends WarnException { + + public AppConnWarnException(int errCode, String desc) { + super(errCode, desc); + } + + public AppConnWarnException(int errCode, String desc, Throwable cause) { + super(errCode, desc); + initCause(cause); + } +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlyDevelopmentAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlyDevelopmentAppConn.java new file mode 100644 index 000000000..fadc9f02a --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlyDevelopmentAppConn.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.ext; + + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; + +/** + * Only the third level standard + * */ +public interface OnlyDevelopmentAppConn extends AppConn { + + DevelopmentIntegrationStandard getOrCreateDevelopmentStandard(); + +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlySSOAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlySSOAppConn.java new file mode 100644 index 000000000..97f17abd1 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlySSOAppConn.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.ext; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard; + +public interface OnlySSOAppConn extends AppConn { + + SSOIntegrationStandard getOrCreateSSOStandard(); + +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlyStructureAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlyStructureAppConn.java new file mode 100644 index 000000000..714c835d2 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/OnlyStructureAppConn.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.ext; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.standard.app.structure.StructureIntegrationStandard; + +public interface OnlyStructureAppConn extends AppConn { + + StructureIntegrationStandard getOrCreateStructureStandard(); + +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/SecondlyAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/SecondlyAppConn.java new file mode 100644 index 000000000..9405f1926 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/SecondlyAppConn.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.ext; + + +public interface SecondlyAppConn extends OnlySSOAppConn, OnlyStructureAppConn { + +} \ No newline at end of file diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/ThirdlyAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/ThirdlyAppConn.java new file mode 100644 index 000000000..d73396949 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/ext/ThirdlyAppConn.java @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.ext; + +/** + * Including the first, second and third level standards + * */ +public interface ThirdlyAppConn extends OnlySSOAppConn, OnlyStructureAppConn, OnlyDevelopmentAppConn{ +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/impl/AbstractAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/impl/AbstractAppConn.java new file mode 100644 index 000000000..062ea5024 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/impl/AbstractAppConn.java @@ -0,0 +1,106 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.impl; + + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnErrorException; +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnWarnException; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlySSOAppConn; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.core.AppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppDesc; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; +import java.util.stream.Collectors; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public abstract class AbstractAppConn implements AppConn { + + private AppDesc appDesc; + private List appStandards; + protected final List appStandardMethodHeader = Arrays.asList("create", "getOrCreate", "get"); + protected final Logger logger = LoggerFactory.getLogger(getClass()); + + @Override + public List getAppStandards() { + if(appStandards == null) { + synchronized (appStandardMethodHeader) { + if(appStandards == null) { + try { + init(); + } catch (AppConnErrorException e) { + throw new AppConnWarnException(e.getErrCode(), e.getMessage(), e); + } + } + } + } + return appStandards; + } + + protected abstract void initialize(); + + /** + * Specification: each appconn needs to define a method starting with create + * and returning as standard type to initialize the specifications owned by appconn. + * */ + @Override + public final void init() throws AppConnErrorException { + initialize(); + appStandards = Arrays.stream(getClass().getDeclaredMethods()).map(method -> { + String methodName = method.getName(); + if(appStandardMethodHeader.stream().anyMatch(methodName::startsWith) && + AppStandard.class.isAssignableFrom(method.getReturnType())) { + try { + return (AppStandard) method.invoke(this); + } catch (ReflectiveOperationException e) { + logger.warn(methodName + " execute failed, ignore to set it into appStandardList of " + getClass().getSimpleName(), e); + } + } + return null; + }).filter(Objects::nonNull).collect(Collectors.toList()); + if(this instanceof OnlySSOAppConn) { + SSORequestService ssoRequestService = ((OnlySSOAppConn) this).getOrCreateSSOStandard().getSSORequestService(); + for(AppStandard appStandard : appStandards) { + if(appStandard instanceof AppIntegrationStandard) { + ((AppIntegrationStandard) appStandard).setSSORequestService(ssoRequestService); + } + try { + appStandard.init(); + } catch (AppStandardErrorException e) { + throw new AppConnErrorException(e.getErrCode(), "Init " + appStandard.getStandardName() + " failed!", e); + } + } + } + } + + @Override + public AppDesc getAppDesc() { + return appDesc; + } + + @Override + public void setAppDesc(AppDesc appDesc) { + this.appDesc = appDesc; + } + +} diff --git a/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/impl/AbstractOnlySSOAppConn.java b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/impl/AbstractOnlySSOAppConn.java new file mode 100644 index 000000000..53ecf8611 --- /dev/null +++ b/dss-appconn/dss-appconn-core/src/main/java/com/webank/wedatasphere/dss/appconn/core/impl/AbstractOnlySSOAppConn.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.core.impl; + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlySSOAppConn; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandardFactory; + +public abstract class AbstractOnlySSOAppConn extends AbstractAppConn implements OnlySSOAppConn { + + private static final SSOIntegrationStandard SSO_INTEGRATION_STANDARD; + + static { + SSOIntegrationStandardFactory ssoIntegrationStandardFactory = ClassUtils.getInstanceOrWarn(SSOIntegrationStandardFactory.class); + ssoIntegrationStandardFactory.init(); + SSO_INTEGRATION_STANDARD = ssoIntegrationStandardFactory.getSSOIntegrationStandard(); + } + + @Override + public final SSOIntegrationStandard getOrCreateSSOStandard() { + return SSO_INTEGRATION_STANDARD; + } +} diff --git a/dss-appconn/dss-appconn-loader/pom.xml b/dss-appconn/dss-appconn-loader/pom.xml new file mode 100644 index 000000000..db9b9e823 --- /dev/null +++ b/dss-appconn/dss-appconn-loader/pom.xml @@ -0,0 +1,86 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-appconn-loader + + + + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + **/*.yml + **/*.properties + **/*.sh + **/log4j2.xml + + + + + + + src/main/resources + + + + + \ No newline at end of file diff --git a/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/clazzloader/AppConnClassLoader.java b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/clazzloader/AppConnClassLoader.java new file mode 100644 index 000000000..7f2c60882 --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/clazzloader/AppConnClassLoader.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.clazzloader; + +import java.net.URL; +import java.net.URLClassLoader; + +/** + * use UrlClassLoader load Jar package to jvm. + * */ +public class AppConnClassLoader extends URLClassLoader { + + public AppConnClassLoader(URL[] urls, ClassLoader parent) { + super(urls, parent); + } + + @Override + public Class loadClass(String name) throws ClassNotFoundException { + return loadClass(name, false); + } + + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + return super.loadClass(name, resolve); + } +} \ No newline at end of file diff --git a/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/AppConnLoader.java b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/AppConnLoader.java new file mode 100644 index 000000000..95df46e5c --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/AppConnLoader.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.loader; + + +import com.webank.wedatasphere.dss.appconn.core.AppConn; + +/** + * Load interface specification of appconn + * */ +public interface AppConnLoader { + + AppConn getAppConn(String appConnName, String spi, String homePath) throws Exception; + +} \ No newline at end of file diff --git a/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/AppConnLoaderFactory.java b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/AppConnLoaderFactory.java new file mode 100644 index 000000000..c84f592c3 --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/AppConnLoaderFactory.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.loader; + + +import com.webank.wedatasphere.dss.appconn.loader.conf.AppConnLoaderConf; +import org.apache.commons.lang.ClassUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class AppConnLoaderFactory { + + private static final Logger logger = LoggerFactory.getLogger(AppConnLoaderFactory.class); + + private static Class clazz = CommonAppConnLoader.class; + private static AppConnLoader appConnLoader = null; + + @SuppressWarnings("unchecked") + public static AppConnLoader getAppConnLoader(){ + if (appConnLoader == null){ + synchronized (AppConnLoaderFactory.class){ + if (appConnLoader == null){ + // The corresponding classes can be loaded by configuration + String className = AppConnLoaderConf.CLASS_LOADER_CLASS_NAME().getValue(); + if (StringUtils.isNotBlank(className)){ + try{ + clazz = ClassUtils.getClass(className); + }catch(ClassNotFoundException e){ + logger.warn(String.format("Can not get AppConnLoader class %s, CommonAppConnLoader will be used by default.", className), e); + } + } + try { + appConnLoader = clazz.newInstance(); + } catch (Exception e) { + logger.error(String.format("Can not initialize AppConnLoader class %s.", clazz.getSimpleName()), e); + } + logger.info("Use {} to load all AppConns.", clazz.getSimpleName()); + } + } + } + return appConnLoader; + } + +} diff --git a/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/CommonAppConnLoader.java b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/CommonAppConnLoader.java new file mode 100644 index 000000000..e47a273a0 --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/loader/CommonAppConnLoader.java @@ -0,0 +1,82 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.loader; + + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.loader.clazzloader.AppConnClassLoader; +import com.webank.wedatasphere.dss.appconn.loader.exception.NoSuchAppConnException; +import com.webank.wedatasphere.dss.appconn.loader.utils.AppConnUtils; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.standard.common.utils.AppStandardClassUtils; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import java.io.File; +import java.net.URL; +import java.nio.file.Paths; +import java.util.List; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class CommonAppConnLoader implements AppConnLoader { + + private static final String LIB_NAME = "lib"; + + private static final Logger LOGGER = LoggerFactory.getLogger(CommonAppConnLoader.class); + + @Override + public AppConn getAppConn(String appConnName, String spi, String homePath) throws Exception { + ClassLoader currentClassLoader = Thread.currentThread().getContextClassLoader(); + String libPathUrl; + if (StringUtils.isNotEmpty(homePath)){ + libPathUrl = new File(homePath, LIB_NAME).getPath(); + } else { + libPathUrl = Paths.get(AppConnUtils.getAppConnHomePath(), appConnName, LIB_NAME).toFile().getPath(); + } + LOGGER.info("The libPath url of AppConn {} is {}.", appConnName, libPathUrl); + List jars = AppConnUtils.getJarsUrlsOfPath(libPathUrl); + ClassLoader classLoader = AppStandardClassUtils.getClassLoader(appConnName, () -> new AppConnClassLoader(jars.toArray(new URL[1]), currentClassLoader)); + Thread.currentThread().setContextClassLoader(classLoader); + String fullClassName; + if (StringUtils.isEmpty(spi)) { + try { + fullClassName = AppConnUtils.getAppConnClassName(appConnName, libPathUrl, classLoader); + } catch (NoSuchAppConnException e) { + Thread.currentThread().setContextClassLoader(currentClassLoader); + throw e; + } + } else { + fullClassName = spi; + } + Class clazz = null; + try { + clazz = classLoader.loadClass(fullClassName); + } catch (ClassNotFoundException e) { + Thread.currentThread().setContextClassLoader(currentClassLoader); + DSSExceptionUtils.dealErrorException(70062, fullClassName + " class not found ", e, ErrorException.class); + } + if (clazz == null) { + Thread.currentThread().setContextClassLoader(currentClassLoader); + return null; + } else { + AppConn retAppConn = (AppConn) clazz.newInstance(); + Thread.currentThread().setContextClassLoader(currentClassLoader); + LOGGER.info("AppConn is {}, retAppConn is {}.", appConnName, retAppConn.getClass().getName()); + return retAppConn; + } + } +} diff --git a/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/utils/AppConnUtils.java b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/utils/AppConnUtils.java new file mode 100644 index 000000000..db12d3d1b --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/utils/AppConnUtils.java @@ -0,0 +1,137 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.utils; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.loader.exception.NoSuchAppConnException; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; +import java.io.File; +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.jar.JarEntry; +import java.util.jar.JarFile; +import org.apache.commons.lang.StringUtils; + +public class AppConnUtils { + + public static final String JAR_SUF_NAME = ".jar"; + + public static final String APPCONN_DIR_NAME = "dss-appconns"; + + public static final CommonVars APPCONN_HOME_PATH = CommonVars.apply("wds.dss.appconn.home.path", + new File(DSSCommonUtils.DSS_HOME.getValue(), APPCONN_DIR_NAME).getPath()); + + public static String getAppConnHomePath() { + return APPCONN_HOME_PATH.acquireNew(); + } + + /** + * Obtain the fully qualified name of the appconn to be instantiated. + * */ + public static String getAppConnClassName(String appConnName, String libPath, + ClassLoader classLoader) throws NoSuchAppConnException, IOException { + //1.Get all the jar packages under the directory + List jars = getJarsOfPath(libPath); + //2.Get the subclass of appconn from all jars + for (String jar : jars) { + for (String clazzName : getClassNameFrom(jar)) { + //3.Then find the subclass of appconn in the corresponding jar package + if (isChildClass(clazzName, AppConn.class, classLoader)) { + return clazzName; + } + } + } + throw new NoSuchAppConnException("Cannot find a appConn instance for AppConn " + appConnName + " in lib path " + libPath); + } + + public static List getJarsOfPath(String path) { + File file = new File(path); + List jars = new ArrayList<>(); + if (file.listFiles() != null) { + for (File f : file.listFiles()) { + // only search from dss-xxxxx.jar. + if (!f.isDirectory() && f.getName().endsWith(JAR_SUF_NAME) && f.getName().startsWith("dss-")) { + jars.add(f.getPath()); + } + } + } + return jars; + } + + + public static List getJarsUrlsOfPath(String path) throws MalformedURLException { + File file = new File(path); + List jars = new ArrayList<>(); + if (file.listFiles() != null) { + for (File f : file.listFiles()) { + if (!f.isDirectory() && f.getName().endsWith(JAR_SUF_NAME)) { + jars.add(f.toURI().toURL()); + } + } + } + return jars; + } + + + /** + * Then look for the subclass of appconn in the corresponding jar package, + * and read all the class file names from the jar package. + */ + private static List getClassNameFrom(String jarName) throws IOException { + List fileList = new ArrayList<>(); + JarFile jarFile = new JarFile(new File(jarName)); + Enumeration en = jarFile.entries(); + while (en.hasMoreElements()) { + String name1 = en.nextElement().getName(); + if (!name1.endsWith(".class")) { + continue; + } + String name2 = name1.substring(0, name1.lastIndexOf(".class")); + String name3 = name2.replaceAll("/", "."); + fileList.add(name3); + } + return fileList; + } + + + private static boolean isChildClass(String className, Class parentClazz, ClassLoader classLoader) { + if (StringUtils.isEmpty(className)) { + return false; + } + Class clazz = null; + try { + clazz = classLoader.loadClass(className); + //忽略抽象类和接口 + if (Modifier.isAbstract(clazz.getModifiers())) { + return false; + } + if (Modifier.isInterface(clazz.getModifiers())) { + return false; + } + } catch (Throwable t) { + return false; + } + return parentClazz.isAssignableFrom(clazz); + } + +} diff --git a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/utils/ExceptionHelper.java b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/utils/ExceptionHelper.java similarity index 82% rename from dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/utils/ExceptionHelper.java rename to dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/utils/ExceptionHelper.java index 39a80227a..69f92a451 100644 --- a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/utils/ExceptionHelper.java +++ b/dss-appconn/dss-appconn-loader/src/main/java/com/webank/wedatasphere/dss/appconn/loader/utils/ExceptionHelper.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,14 +14,10 @@ * */ -package com.webank.wedatasphere.dss.appjoint.utils; +package com.webank.wedatasphere.dss.appconn.loader.utils; import com.webank.wedatasphere.linkis.common.exception.ErrorException; -/** - * created by cooperyang on 2019/11/10 - * Description: - */ public class ExceptionHelper { public static void dealErrorException(int errorCode, String errorMsg, Throwable t) throws ErrorException { ErrorException errorException = new ErrorException(errorCode, errorMsg); diff --git a/dss-appconn/dss-appconn-loader/src/main/scala/com/webank/wedatasphere/dss/appconn/loader/conf/AppConnLoaderConf.scala b/dss-appconn/dss-appconn-loader/src/main/scala/com/webank/wedatasphere/dss/appconn/loader/conf/AppConnLoaderConf.scala new file mode 100644 index 000000000..9c4942c2a --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/scala/com/webank/wedatasphere/dss/appconn/loader/conf/AppConnLoaderConf.scala @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.conf + +import com.webank.wedatasphere.linkis.common.conf.CommonVars + +object AppConnLoaderConf { + val CLASS_LOADER_CLASS_NAME = CommonVars("dss.appconn.loader.classname", "") +} diff --git a/dss-appconn/dss-appconn-loader/src/main/scala/com/webank/wedatasphere/dss/appconn/loader/exception/NoSuchAppConnException.scala b/dss-appconn/dss-appconn-loader/src/main/scala/com/webank/wedatasphere/dss/appconn/loader/exception/NoSuchAppConnException.scala new file mode 100644 index 000000000..a590d77a3 --- /dev/null +++ b/dss-appconn/dss-appconn-loader/src/main/scala/com/webank/wedatasphere/dss/appconn/loader/exception/NoSuchAppConnException.scala @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.loader.exception + +import com.webank.wedatasphere.linkis.common.exception.ErrorException + +case class NoSuchAppConnException(errDesc:String) extends ErrorException(70059, errDesc) + diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/pom.xml b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/pom.xml new file mode 100644 index 000000000..fcbddd097 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/pom.xml @@ -0,0 +1,88 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-appconn-manager-client + + + + com.webank.wedatasphere.dss + dss-appconn-manager-core + ${dss.version} + + + com.webank.wedatasphere.linkis + linkis-bml-client + ${linkis.version} + + + commons-beanutils + commons-beanutils + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + **/*.yml + **/*.properties + **/*.sh + **/log4j2.xml + + + + + + + src/main/resources + + + + + \ No newline at end of file diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/conf/AppConnManagerClientConfiguration.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/conf/AppConnManagerClientConfiguration.java new file mode 100644 index 000000000..fbf8889e0 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/conf/AppConnManagerClientConfiguration.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +public class AppConnManagerClientConfiguration { + + public final static CommonVars DSS_APPCONN_CLIENT_TOKEN = CommonVars.apply("wds.dss.appconn.client.user.token","WS-AUTH"); + public final static CommonVars LINKIS_ADMIN_USER = CommonVars.apply("wds.dss.appconn.client.user","ws"); + + + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppConnInfoImpl.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppConnInfoImpl.java new file mode 100644 index 000000000..54ff05d9a --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppConnInfoImpl.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.entity; + +import com.webank.wedatasphere.dss.common.entity.Resource; + +public class AppConnInfoImpl implements AppConnInfo { + + private String appConnName; + private String className; + private Resource appConnResource; + + public void setAppConnName(String appConnName) { + this.appConnName = appConnName; + } + + public void setClassName(String className) { + this.className = className; + } + + public void setAppConnResource(Resource appConnResource) { + this.appConnResource = appConnResource; + } + + @Override + public String getAppConnName() { + return appConnName; + } + + @Override + public String getClassName() { + return className; + } + + @Override + public Resource getAppConnResource() { + return appConnResource; + } +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppInstanceInfoImpl.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppInstanceInfoImpl.java new file mode 100644 index 000000000..fb08509a8 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppInstanceInfoImpl.java @@ -0,0 +1,81 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.entity; + +public class AppInstanceInfoImpl implements AppInstanceInfo { + + private Long id; + private String url; + private String homepageUrl; + private String redirectUrl; + private String labels; + private String enhanceJson; + + @Override + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + @Override + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + @Override + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + @Override + public String getRedirectUrl() { + return redirectUrl; + } + + public void setRedirectUrl(String redirectUrl) { + this.redirectUrl = redirectUrl; + } + + @Override + public String getLabels() { + return labels; + } + + public void setLabels(String labels) { + this.labels = labels; + } + + @Override + public String getEnhanceJson() { + return enhanceJson; + } + + public void setEnhanceJson(String enhanceJson) { + this.enhanceJson = enhanceJson; + } +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/exception/AppConnHomeNotExistsWarnException.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/exception/AppConnHomeNotExistsWarnException.java new file mode 100644 index 000000000..441aca022 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/exception/AppConnHomeNotExistsWarnException.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.exception; + +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; + + +public class AppConnHomeNotExistsWarnException extends DSSRuntimeException { + + public AppConnHomeNotExistsWarnException(int errCode, String desc) { + super(errCode, desc); + } + + public AppConnHomeNotExistsWarnException(int errCode, String desc, Throwable t) { + super(errCode, desc); + initCause(t); + } +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnInfoServiceImpl.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnInfoServiceImpl.java new file mode 100644 index 000000000..b91f3d49c --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnInfoServiceImpl.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.service; + +import com.webank.wedatasphere.dss.appconn.manager.action.GetAppConnInfoAction; +import com.webank.wedatasphere.dss.appconn.manager.action.GetAppConnInfosAction; +import com.webank.wedatasphere.dss.appconn.manager.action.GetAppInstancesAction; +import com.webank.wedatasphere.dss.appconn.manager.conf.AppConnManagerClientConfiguration; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppInstanceInfo; +import com.webank.wedatasphere.dss.appconn.manager.result.GetAppConnInfoResult; +import com.webank.wedatasphere.dss.appconn.manager.result.GetAppConnInfosResult; +import com.webank.wedatasphere.dss.appconn.manager.result.GetAppInstancesResult; +import com.webank.wedatasphere.linkis.common.conf.Configuration; +import com.webank.wedatasphere.linkis.httpclient.config.ClientConfig; +import com.webank.wedatasphere.linkis.httpclient.dws.DWSHttpClient; +import com.webank.wedatasphere.linkis.httpclient.dws.authentication.TokenAuthenticationStrategy; +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfig; +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfigBuilder; +import java.util.List; + +import static com.webank.wedatasphere.dss.appconn.manager.conf.AppConnManagerClientConfiguration.LINKIS_ADMIN_USER; + +public class AppConnInfoServiceImpl implements AppConnInfoService { + + private ClientConfig clientConfig = DWSClientConfigBuilder.newBuilder().setDWSVersion("v1").addServerUrl(Configuration.getGateWayURL()) + .connectionTimeout(300000).discoveryEnabled(false).setAuthenticationStrategy(new TokenAuthenticationStrategy()).setAuthTokenKey(LINKIS_ADMIN_USER.getValue()) + .setAuthTokenValue(AppConnManagerClientConfiguration.DSS_APPCONN_CLIENT_TOKEN.getValue()).maxConnectionSize(50).readTimeout(300000).build(); + private DWSHttpClient client = new DWSHttpClient((DWSClientConfig) clientConfig, "AppConn-Client-"); + + @Override + public List getAppConnInfos() { + GetAppConnInfosAction getAppConnInfosAction =new GetAppConnInfosAction(); + getAppConnInfosAction.setUser(LINKIS_ADMIN_USER.getValue()); + GetAppConnInfosResult result = (GetAppConnInfosResult) client.execute(getAppConnInfosAction); + return result.getAppConnInfos(); + } + + @Override + public AppConnInfo getAppConnInfo(String appConnName) { + GetAppConnInfoAction action = new GetAppConnInfoAction(); + action.setAppConnName(appConnName); + action.setUser(LINKIS_ADMIN_USER.getValue()); + GetAppConnInfoResult result = (GetAppConnInfoResult) client.execute(action); + return result.getAppConnInfo(); + } + + @Override + public List getAppInstancesByAppConnInfo(AppConnInfo appConnInfo) { + return getAppInstancesByAppConnName(appConnInfo.getAppConnName()); + } + + @Override + public List getAppInstancesByAppConnName(String appConnName) { + GetAppInstancesAction action = new GetAppInstancesAction(); + action.setAppConnName(appConnName); + action.setUser(LINKIS_ADMIN_USER.getValue()); + GetAppInstancesResult result = (GetAppInstancesResult) client.execute(action); + return result.getAppInstanceInfos(); + } +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnResourceServiceImpl.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnResourceServiceImpl.java new file mode 100644 index 000000000..edfd2c5fa --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnResourceServiceImpl.java @@ -0,0 +1,123 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.service; + +import com.webank.wedatasphere.dss.appconn.loader.utils.AppConnUtils; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.exception.AppConnHomeNotExistsWarnException; +import com.webank.wedatasphere.dss.appconn.manager.impl.AbstractAppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.utils.AppConnIndexFileUtils; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.ZipHelper; +import com.webank.wedatasphere.linkis.bml.client.BmlClient; +import com.webank.wedatasphere.linkis.bml.client.BmlClientFactory; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.IntStream; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class AppConnResourceServiceImpl implements AppConnResourceService { + + private static final Logger LOGGER = LoggerFactory.getLogger(AppConnResourceServiceImpl.class); + + private BmlClient bmlClient = BmlClientFactory.createBmlClient(); + + @Override + public String getAppConnHome(AppConnInfo appConnInfo) { + File appConnHomePath = new File(AppConnUtils.getAppConnHomePath()); + String appConnName = appConnInfo.getAppConnName(); + if(!appConnHomePath.exists()) { + throw new AppConnHomeNotExistsWarnException(20350, "AppConn home path " + appConnHomePath.getPath() + " not exists."); + } else if (!appConnHomePath.isDirectory()) { + throw new AppConnHomeNotExistsWarnException(20350, "AppConn home path " + appConnHomePath.getPath() + " is not a directory."); + } + File appConnPath = new File(appConnHomePath, appConnName); + Resource resource = appConnInfo.getAppConnResource(); + if(!appConnPath.exists() && !appConnPath.mkdir()) { + throw new AppConnHomeNotExistsWarnException(20350, "Cannot create dir " + appConnPath.getPath() + " for AppConn " + appConnName); + } + if(AppConnIndexFileUtils.isLatestIndex(appConnPath, resource)) { + return appConnPath.getPath(); + } + LOGGER.info("Try to download latest resource {} in version {} from BML for AppConn {}.", resource.getResourceId(), + resource.getVersion(), appConnName); + // At first, Download AppConn files from bml. + String zipFilePath = new File(appConnHomePath, appConnName + "_" + resource.getVersion() + ".zip").getPath(); + bmlClient.downloadResource(Utils.getJvmUser(), resource.getResourceId(), resource.getVersion(), + "file://" + zipFilePath, true); + // Then, try to unzip it. + if(!DeleteAppConnDir(appConnPath)) { + throw new AppConnHomeNotExistsWarnException(20350, "Cannot delete dir " + appConnPath.getPath() + " for AppConn " + appConnName); + } + try { + ZipHelper.unzip(zipFilePath); + } catch (DSSErrorException e) { + throw new AppConnHomeNotExistsWarnException(20350, "Unzip " + zipFilePath + " failed, AppConn " + appConnName, e); + } + + File oldIndexFile = AppConnIndexFileUtils.getIndexFile(appConnPath); + + // update index file. + if (oldIndexFile != null && !oldIndexFile.delete()) { + throw new AppConnHomeNotExistsWarnException(20350, "Delete index file " + oldIndexFile.getName() + " failed, please ensure the permission is all right."); + } + +// TODO ZipUtils.fileToUnzip(zipFilePath, appConnHomePath.getPath()); + // Only reserve latest 2 version files. + File[] historyZipFiles = appConnHomePath.listFiles((p, fileName) -> fileName.startsWith(appConnName) && fileName.endsWith(".zip")); + if(historyZipFiles.length > 2) { + List files = Arrays.stream(historyZipFiles).sorted().collect(Collectors.toList()); + // ignore delete failed. + IntStream.range(0, files.size() - 2).forEach(index -> files.get(index).delete()); + } + // Finally, reload appconn and write index file. + Path indexFile = Paths.get(appConnPath.getPath(), AppConnIndexFileUtils.getIndexFileName(resource)); + try { + Files.createFile(indexFile); + } catch (IOException e) { + throw new AppConnHomeNotExistsWarnException(20350, "Cannot create index file " + indexFile.toFile().getPath() + " for AppConn " + + appConnName, e); + } + ((AbstractAppConnManager) AppConnManager.getAppConnManager()).reloadAppConn(appConnInfo); + return appConnPath.getPath(); + } + + public boolean DeleteAppConnDir(File f){ + if(f.isDirectory()){ + File[] files = f.listFiles(); + for (File key : files) { + if(key.isFile()){ + key.delete(); + }else{ + DeleteAppConnDir(key); + } + } + } + return f.delete(); + } + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/action/GetAppConnInfosAction.scala b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/action/GetAppConnInfosAction.scala new file mode 100644 index 000000000..6d7ac5961 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/action/GetAppConnInfosAction.scala @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.action + + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.{GetAction, UserAction} + +class GetAppConnInfosAction extends GetAction with DWSHttpAction with UserAction{ + + private var user:String = _ + + override def suffixURLs: Array[String] = Array("dss", "framework", "project", "appconn", "listAppConnInfos") + + override def setUser(user: String): Unit = this.user = user + + override def getUser: String = this.user +} + +class GetAppConnInfoAction extends GetAction with DWSHttpAction with UserAction { + + private var appConnName: String = _ + private var user:String = _ + + override def suffixURLs: Array[String] = Array("dss", "framework", "project", "appconn", appConnName, "get") + + def setAppConnName(appConnName: String): Unit = this.appConnName = appConnName + + override def setUser(user: String): Unit = this.user = user + + override def getUser: String = this.user + +} \ No newline at end of file diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/action/GetAppInstancesAction.scala b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/action/GetAppInstancesAction.scala new file mode 100644 index 000000000..16ffd447d --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/action/GetAppInstancesAction.scala @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.action + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.{GetAction, UserAction} + +class GetAppInstancesAction extends GetAction with DWSHttpAction with UserAction { + + private var user:String = _ + + private var appConnName: String = _ + + override def suffixURLs: Array[String] = Array("dss", "framework", "project", "appconn", appConnName, "getAppInstances") + + def setAppConnName(appConnName: String): Unit = this.appConnName = appConnName + + override def setUser(user: String): Unit = this.user = user + + override def getUser: String = this.user + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppConnInfoResult.scala b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppConnInfoResult.scala new file mode 100644 index 000000000..89147e419 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppConnInfoResult.scala @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.result + +import java.util + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfoImpl +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/framework/project/appconn/[a-zA-Z0-9]+/get") +class GetAppConnInfoResult extends DWSResult { + + private var appConnInfo: AppConnInfoImpl = _ + + def setAppConnInfo(appConnInfo: util.Map[String, Object]): Unit = { + val json = DSSCommonUtils.COMMON_GSON.toJson(appConnInfo) + this.appConnInfo = DSSCommonUtils.COMMON_GSON.fromJson(json, classOf[AppConnInfoImpl]) + } + + def getAppConnInfo: AppConnInfoImpl = appConnInfo + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppConnInfosResult.scala b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppConnInfosResult.scala new file mode 100644 index 000000000..8a3dd2f95 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppConnInfosResult.scala @@ -0,0 +1,43 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.result + +import java.util + +import com.webank.wedatasphere.dss.appconn.manager.entity.{AppConnInfo, AppConnInfoImpl} +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +import scala.collection.convert.wrapAsScala._ +import scala.collection.convert.wrapAsJava._ + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/framework/project/appconn/listAppConnInfos") +class GetAppConnInfosResult extends DWSResult { + + private var appConnInfos: util.List[AppConnInfo] = _ + + def setAppConnInfos(appConnInfos: util.List[util.Map[String, Object]]): Unit = { + this.appConnInfos = appConnInfos.map{ map => + val json = DSSCommonUtils.COMMON_GSON.toJson(map) + DSSCommonUtils.COMMON_GSON.fromJson(json, classOf[AppConnInfoImpl]).asInstanceOf[AppConnInfo] + } + } + + def getAppConnInfos: util.List[AppConnInfo] = appConnInfos + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppInstancesResult.scala b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppInstancesResult.scala new file mode 100644 index 000000000..9f0113fc5 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-client/src/main/scala/com/webank/wedatasphere/dss/appconn/manager/result/GetAppInstancesResult.scala @@ -0,0 +1,43 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.result + +import java.util + +import com.webank.wedatasphere.dss.appconn.manager.entity.{AppInstanceInfo, AppInstanceInfoImpl} +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +import scala.collection.convert.wrapAsJava._ +import scala.collection.convert.wrapAsScala._ + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/framework/project/appconn/[a-zA-Z0-9\\-]+/getAppInstances") +class GetAppInstancesResult extends DWSResult { + + private var appInstanceInfos: util.List[AppInstanceInfo] = _ + + def setAppInstanceInfos(appInstanceInfos: util.List[util.Map[String, Object]]): Unit = { + this.appInstanceInfos = appInstanceInfos.map{ map => + val json = DSSCommonUtils.COMMON_GSON.toJson(map) + DSSCommonUtils.COMMON_GSON.fromJson(json, classOf[AppInstanceInfoImpl]) + } + } + + def getAppInstanceInfos: util.List[AppInstanceInfo] = appInstanceInfos + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/pom.xml b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/pom.xml new file mode 100644 index 000000000..d1629597c --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/pom.xml @@ -0,0 +1,73 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-appconn-manager-core + jar + + + + com.webank.wedatasphere.dss + dss-appconn-loader + ${dss.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + -target:jvm-1.8 + + + + org.apache.maven.plugins + maven-jar-plugin + + + **/*.yml + **/*.properties + **/*.sh + **/log4j2.xml + + + + + + + src/main/resources + + + + + \ No newline at end of file diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/AppConnManager.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/AppConnManager.java new file mode 100644 index 000000000..b4a797d70 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/AppConnManager.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnErrorException; +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnWarnException; +import com.webank.wedatasphere.dss.appconn.manager.impl.AbstractAppConnManager; +import java.util.List; +import java.util.stream.Collectors; + + +public interface AppConnManager { + + /** + * init AppConnManager + */ + void init(); + + /** + * get all AppConns + */ + List listAppConns(); + + default List listAppConns(Class appConnClass) { + return listAppConns().stream().filter(appConnClass::isInstance).map(appConn -> (T) appConn).collect(Collectors.toList()); + } + + /** + * Returns an appconn by its name + */ + AppConn getAppConn(String appConnName); + + default T getAppConn(Class appConnClass) { + List appConns = listAppConns().stream().filter(appConnClass::isInstance).collect(Collectors.toList()); + if(appConns.isEmpty()) { + throw new AppConnWarnException(25344, "Cannot find a AppConn instance for " + appConnClass.getSimpleName()); + } else if(appConns.size() > 1) { + throw new AppConnWarnException(25344, "More than one AppConn instances is found, list: " + appConns); + } + return (T) appConns.get(0); + } + + void reloadAppConn(String appConnName); + + static void setLazyLoad() { + AbstractAppConnManager.setLazyLoad(); + } + + static AppConnManager getAppConnManager() { + return AbstractAppConnManager.getAppConnManager(); + } + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppConnInfo.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppConnInfo.java new file mode 100644 index 000000000..af3e4eebd --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppConnInfo.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.entity; + +import com.webank.wedatasphere.dss.common.entity.Resource; + +public interface AppConnInfo { + + String getAppConnName(); + + String getClassName(); + + Resource getAppConnResource(); + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppInstanceInfo.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppInstanceInfo.java new file mode 100644 index 000000000..3ee07e601 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/entity/AppInstanceInfo.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.entity; + +public interface AppInstanceInfo { + + Long getId(); + + String getUrl(); + + String getHomepageUrl(); + + String getRedirectUrl(); + + String getLabels(); + + String getEnhanceJson(); + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/exception/AppConnIndexFileWarnException.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/exception/AppConnIndexFileWarnException.java new file mode 100644 index 000000000..75884f42e --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/exception/AppConnIndexFileWarnException.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.exception; + +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; + +public class AppConnIndexFileWarnException extends DSSRuntimeException { + + public AppConnIndexFileWarnException(int errCode, String desc) { + super(errCode, desc); + } +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/impl/AbstractAppConnManager.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/impl/AbstractAppConnManager.java new file mode 100644 index 000000000..e7661812c --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/impl/AbstractAppConnManager.java @@ -0,0 +1,212 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.impl; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.loader.loader.AppConnLoader; +import com.webank.wedatasphere.dss.appconn.loader.loader.AppConnLoaderFactory; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppInstanceInfo; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnInfoService; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnResourceService; +import com.webank.wedatasphere.dss.appconn.manager.utils.AppInstanceConstants; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.standard.common.desc.AppDescImpl; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstanceImpl; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public abstract class AbstractAppConnManager implements AppConnManager { + + private static final Logger LOGGER = LoggerFactory.getLogger(AbstractAppConnManager.class); + private final AppConnLoader appConnLoader = AppConnLoaderFactory.getAppConnLoader(); + + private final Map appConns = new HashMap<>(); + private boolean isLoaded = false; + private List appConnList = null; + private AppConnInfoService appConnInfoService; + private AppConnResourceService appConnResourceService; + + private static AppConnManager appConnManager; + private static boolean lazyLoad = false; + + public static void setLazyLoad() { + lazyLoad = true; + } + + public static AppConnManager getAppConnManager() { + if(appConnManager != null) { + return appConnManager; + } + synchronized (AbstractAppConnManager.class) { + if(appConnManager == null) { + appConnManager = ClassUtils.getInstanceOrDefault(AppConnManager.class, new AppConnManagerImpl()); + LOGGER.info("The instance of AppConnManager is {}.", appConnManager.getClass().getName()); + appConnManager.init(); + } + } + return appConnManager; + } + + @Override + public void init() { + appConnInfoService = createAppConnInfoService(); + LOGGER.info("The instance of AppConnInfoService is {}.", appConnInfoService.getClass().getName()); + appConnResourceService = createAppConnResourceService(); + LOGGER.info("The instance of AppConnResourceService is {}.", appConnResourceService.getClass().getName()); + if(!lazyLoad && !isLoaded) { + loadAppConns(); + isLoaded = true; + } + } + + protected abstract AppConnInfoService createAppConnInfoService(); + + protected abstract AppConnResourceService createAppConnResourceService(); + + protected void loadAppConns() { + LOGGER.info("Begin to init all AppConns."); + List appConnInfos = appConnInfoService.getAppConnInfos(); + if(appConnInfos == null || appConnInfos.isEmpty()) { + if(appConns.isEmpty()) { + throw new DSSRuntimeException("No AppConnInfos returned when the first time init AppConnList."); + } + LOGGER.warn("No AppConnInfos returned, ignore it."); + return; + } + Map appConns = new HashMap<>(); + appConnInfos.forEach(DSSExceptionUtils.handling(appConnInfo -> { + AppConn appConn = loadAppConn(appConnInfo); + appConns.put(appConnInfo.getAppConnName(), appConn); + })); + synchronized (this.appConns) { + this.appConns.clear(); + this.appConns.putAll(appConns); + appConnList = Collections.unmodifiableList(new ArrayList<>(appConns.values())); + } + LOGGER.info("Inited all AppConns, the AppConn list are {}.", this.appConnList); + } + + protected AppConn loadAppConn(AppConnInfo appConnInfo) throws Exception { + LOGGER.info("Ready to load AppConn {}, the appConnInfo are {}.", appConnInfo.getAppConnName(), appConnInfo); + String appConnHome = appConnResourceService.getAppConnHome(appConnInfo); + LOGGER.info("Try to load AppConn {} with home path {}.", appConnInfo.getAppConnName(), appConnHome); + AppConn appConn = appConnLoader.getAppConn(appConnInfo.getAppConnName(), + appConnInfo.getClassName(), appConnHome); + appConn.init(); + List instanceInfos = appConnInfoService.getAppInstancesByAppConnInfo(appConnInfo); + LOGGER.info("The instanceInfos of AppConn {} are {}.", appConnInfo.getAppConnName(), instanceInfos); + AppDescImpl appDesc = new AppDescImpl(); + for (AppInstanceInfo instanceBean : instanceInfos) { + AppInstanceImpl appInstance = new AppInstanceImpl(); + copyProperties(appConnInfo.getAppConnName(), instanceBean, appInstance); + appDesc.addAppInstance(appInstance); + } + appDesc.setAppName(appConnInfo.getAppConnName()); + appConn.setAppDesc(appDesc); + LOGGER.info("AppConn {} is loaded successfully.", appConnInfo.getAppConnName()); + return appConn; + } + + private void copyProperties(String appConnName, AppInstanceInfo appInstanceInfo, AppInstanceImpl appInstance){ + appInstance.setBaseUrl(appInstanceInfo.getUrl()); + Map config = new HashMap<>(); + if(StringUtils.isNotEmpty(appInstanceInfo.getEnhanceJson())){ + try{ + config = DSSCommonUtils.COMMON_GSON.fromJson(appInstanceInfo.getEnhanceJson(), Map.class); + } catch (Exception e) { + LOGGER.error("The json of AppConn {} is not a correct json. content: {}.", appConnName, appInstanceInfo.getEnhanceJson(), e); + throw new DSSRuntimeException("The json of AppConn " + appConnName + " is not a correct json."); + } + } + appInstance.setConfig(config); + if(StringUtils.isNotBlank(appInstanceInfo.getRedirectUrl())) { + config.put(AppInstanceConstants.REDIRECT_URL, appInstanceInfo.getRedirectUrl()); + } + if(StringUtils.isNotBlank(appInstanceInfo.getHomepageUrl())) { + config.put(AppInstanceConstants.HOMEPAGE_URL, appInstanceInfo.getHomepageUrl()); + } + //TODO should use Linkis labelFactory to new labels. + List labels = Arrays.stream(appInstanceInfo.getLabels().split(",")).map(EnvDSSLabel::new) + .collect(Collectors.toList()); + appInstance.setLabels(labels); + appInstance.setId(appInstanceInfo.getId()); + } + + private void lazyLoadAppConns() { + if(lazyLoad && !isLoaded) { + synchronized (this.appConns) { + if(lazyLoad && !isLoaded) { + loadAppConns(); + } + } + } + } + + @Override + public List listAppConns() { + lazyLoadAppConns(); + return appConnList; + } + + + @Override + public AppConn getAppConn(String appConnName) { + lazyLoadAppConns(); + return appConns.get(appConnName); + } + + @Override + public void reloadAppConn(String appConnName) { + AppConnInfo appConnInfo = appConnInfoService.getAppConnInfo(appConnName); + if(appConnInfo == null) { + throw new DSSRuntimeException("Cannot get any info about AppConn " + appConnName); + } + reloadAppConn(appConnInfo); + } + + public void reloadAppConn(AppConnInfo appConnInfo) { + lazyLoadAppConns(); + AppConn appConn; + try { + appConn = loadAppConn(appConnInfo); + } catch (Exception e) { + LOGGER.error("Reload AppConn failed.", e); + throw new DSSRuntimeException("Load AppConn " + appConnInfo.getAppConnName() + " failed!"); + } + synchronized (this.appConns) { + this.appConns.put(appConnInfo.getAppConnName(), appConn); + appConnList = Collections.unmodifiableList(new ArrayList<>(appConns.values())); + } + LOGGER.info("Reloaded AppConn {}.", appConnInfo.getAppConnName()); + } + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/impl/AppConnManagerImpl.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/impl/AppConnManagerImpl.java new file mode 100644 index 000000000..c06025d8e --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/impl/AppConnManagerImpl.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.impl; + +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnInfoService; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnResourceService; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; + + +public class AppConnManagerImpl extends AbstractAppConnManager { + + @Override + protected AppConnInfoService createAppConnInfoService() { + try { + return ClassUtils.getInstance(AppConnInfoService.class); + } catch (DSSErrorException e) { + throw new DSSRuntimeException(25000, "Cannot find a useful AppConnInfoService.", e); + } + } + + @Override + protected AppConnResourceService createAppConnResourceService() { + try { + return ClassUtils.getInstance(AppConnResourceService.class); + } catch (DSSErrorException e) { + throw new DSSRuntimeException(25000, "Cannot find a useful AppConnResourceService.", e); + } + } + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnInfoService.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnInfoService.java new file mode 100644 index 000000000..071dfe13b --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnInfoService.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.service; + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppInstanceInfo; +import java.util.List; + + +public interface AppConnInfoService { + + List getAppConnInfos(); + + AppConnInfo getAppConnInfo(String appConnName); + + List getAppInstancesByAppConnInfo(AppConnInfo appConnInfo); + + List getAppInstancesByAppConnName(String appConnName); + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnResourceService.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnResourceService.java new file mode 100644 index 000000000..371f420d5 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/service/AppConnResourceService.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.service; + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; + +public interface AppConnResourceService { + + String getAppConnHome(AppConnInfo appConnInfo); + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/utils/AppConnIndexFileUtils.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/utils/AppConnIndexFileUtils.java new file mode 100644 index 000000000..01c0278be --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/utils/AppConnIndexFileUtils.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.utils; + +import com.webank.wedatasphere.dss.appconn.manager.exception.AppConnIndexFileWarnException; +import com.webank.wedatasphere.dss.common.entity.Resource; +import java.io.File; +import java.util.Arrays; + +import static com.webank.wedatasphere.dss.appconn.manager.utils.AppInstanceConstants.*; + +public class AppConnIndexFileUtils { + + private static boolean isIndexFile(String fileName) { + return fileName.startsWith(INDEX_FILE_PREFIX) && fileName.endsWith(INDEX_FILE_SUFFIX); + } + + public static File getIndexFile(File parent) { + File[] indexFiles = parent.listFiles((p, fileName) -> isIndexFile(fileName)); + if(indexFiles == null) { + return null; + }else if(indexFiles.length > 1) { + throw new AppConnIndexFileWarnException(20533, "More than one index files exists, indexFile list: " + Arrays.asList(indexFiles)); + } else if(indexFiles.length == 0) { + return null; + } else { + return indexFiles[0]; + } + } + + public static boolean isLatestIndex(File parent, Resource resource) { + File indexFile = getIndexFile(parent); + if(indexFile == null) { + return false; + } + String version = indexFile.getName().replaceAll(INDEX_FILE_PREFIX, "").replaceAll(INDEX_FILE_SUFFIX, ""); + return resource.getVersion().equals(version); + } + + public static String getIndexFileName(Resource resource) { + return INDEX_FILE_PREFIX + resource.getVersion() + INDEX_FILE_SUFFIX; + } + +} diff --git a/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/utils/AppInstanceConstants.java b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/utils/AppInstanceConstants.java new file mode 100644 index 000000000..fa3e0a2c7 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/dss-appconn-manager-core/src/main/java/com/webank/wedatasphere/dss/appconn/manager/utils/AppInstanceConstants.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.manager.utils; + +public class AppInstanceConstants { + + public static final String REDIRECT_URL = "redirectUrl"; + public static final String HOMEPAGE_URL = "homepageUrl"; + + static final String INDEX_FILE_PREFIX = "index_"; + static final String INDEX_FILE_SUFFIX = ".index"; + +} diff --git a/dss-appconn/dss-appconn-manager/pom.xml b/dss-appconn/dss-appconn-manager/pom.xml new file mode 100644 index 000000000..2d4189540 --- /dev/null +++ b/dss-appconn/dss-appconn-manager/pom.xml @@ -0,0 +1,36 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-appconn-manager + pom + + + dss-appconn-manager-core + dss-appconn-manager-client + + + \ No newline at end of file diff --git a/dss-appconn/dss-scheduler-appconn/pom.xml b/dss-appconn/dss-scheduler-appconn/pom.xml new file mode 100644 index 000000000..841183fa7 --- /dev/null +++ b/dss-appconn/dss-scheduler-appconn/pom.xml @@ -0,0 +1,74 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-scheduler-appconn + + + + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + jackson-databind + com.fasterxml.jackson.core + + + + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + + + com.webank.wedatasphere.dss + dss-structure-integration-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-workflow-conversion-standard + ${dss.version} + compile + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + \ No newline at end of file diff --git a/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/AbstractSchedulerAppConn.java b/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/AbstractSchedulerAppConn.java new file mode 100644 index 000000000..afaf4456e --- /dev/null +++ b/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/AbstractSchedulerAppConn.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.scheduler; + +import com.webank.wedatasphere.dss.appconn.core.impl.AbstractOnlySSOAppConn; +import com.webank.wedatasphere.dss.workflow.conversion.WorkflowConversionIntegrationStandard; + +public abstract class AbstractSchedulerAppConn extends AbstractOnlySSOAppConn implements SchedulerAppConn { + + private WorkflowConversionIntegrationStandard workflowConversionIntegrationStandard; + + @Override + protected void initialize() { + workflowConversionIntegrationStandard = new WorkflowConversionIntegrationStandard(); + } + + @Override + public WorkflowConversionIntegrationStandard getOrCreateWorkflowConversionStandard() { + workflowConversionIntegrationStandard.setSSORequestService(this.getOrCreateSSOStandard().getSSORequestService()); + workflowConversionIntegrationStandard.setAppConnName(getAppDesc().getAppName()); + return workflowConversionIntegrationStandard; + } +} diff --git a/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/SchedulerAppConn.java b/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/SchedulerAppConn.java new file mode 100644 index 000000000..44f509a9c --- /dev/null +++ b/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/SchedulerAppConn.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.scheduler; + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlySSOAppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyStructureAppConn; +import com.webank.wedatasphere.dss.workflow.conversion.WorkflowConversionIntegrationStandard; + +public interface SchedulerAppConn extends OnlySSOAppConn, OnlyStructureAppConn { + + WorkflowConversionIntegrationStandard getOrCreateWorkflowConversionStandard(); + +} diff --git a/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/conf/SchedulerAppConnConfiguration.java b/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/conf/SchedulerAppConnConfiguration.java new file mode 100644 index 000000000..2ed7b67a5 --- /dev/null +++ b/dss-appconn/dss-scheduler-appconn/src/main/java/com/webank/wedatasphere/dss/appconn/scheduler/conf/SchedulerAppConnConfiguration.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.appconn.scheduler.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +public class SchedulerAppConnConfiguration { + + public static final String SCHEDULER_APP_CONN_NAME = CommonVars.apply("wds.dss.appconn.scheduler.name", "schedulis").getValue(); + +} diff --git a/dss-appconn/linkis-appconn-engineplugin/pom.xml b/dss-appconn/linkis-appconn-engineplugin/pom.xml new file mode 100644 index 000000000..b1b7daa1c --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/pom.xml @@ -0,0 +1,187 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + linkis-appconn-engineplugin + + + + com.webank.wedatasphere.linkis + linkis-engineconn-plugin-core + ${linkis.version} + + + com.webank.wedatasphere.linkis + linkis-computation-engineconn + ${linkis.version} + + + reflections + org.reflections + + + jackson-core + com.fasterxml.jackson.core + + + javassist + org.javassist + + + + + + com.webank.wedatasphere.linkis + linkis-message-scheduler + ${linkis.version} + + + hibernate-validator + org.hibernate.validator + + + provided + + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-scheduler-appconn + ${dss.version} + + + spring-beans + org.springframework + + + + + + com.webank.wedatasphere.dss + dss-appconn-manager-client + ${dss.version} + + + + + com.webank.wedatasphere.dss + dss-development-process-standard-execution + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + + + + javax.mail + mail + 1.4 + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + -target:jvm-1.8 + + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + linkis-engineplugin-appconn + false + false + + src/main/assembly/distribution.xml + + + + + + + ${basedir}/src/main/resources + + **/*.properties + **/*.xml + **/*.yml + + + + ${project.artifactId}-${project.version} + + + \ No newline at end of file diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/assembly/distribution.xml b/dss-appconn/linkis-appconn-engineplugin/src/main/assembly/distribution.xml new file mode 100644 index 000000000..3001d75a1 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/assembly/distribution.xml @@ -0,0 +1,78 @@ + + + + linkis-enginePlugin-appconn + + zip + + true + linkis-engineplugin-appconn + + + + + + /dist/v1/lib + true + true + false + true + true + + + + + + + ${basedir}/src/main/resources + + enginePlugin.properties + + 0777 + / + unix + + + + ${basedir}/src/main/resources + + *.properties + log4j2*.xml + + 0777 + /dist/v1/conf + unix + + + + ${basedir}/target + + *.jar + + + *doc.jar + + /plugin/1 + + + + + + diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/resources/linkis-engineconn.properties b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/linkis-engineconn.properties new file mode 100644 index 000000000..99bba7202 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/linkis-engineconn.properties @@ -0,0 +1,45 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +wds.linkis.server.version=v1 + +wds.linkis.engineconn.spark.conf.enable=true + +wds.linkis.engineconn.debug.enable=true + +#wds.linkis.keytab.enable=true + +wds.linkis.engineconn.plugin.default.class=com.webank.wedatasphere.linkis.manager.engineplugin.appconn.AppConnEngineConnPlugin + +##mybatis +wds.linkis.server.mybatis.mapperLocations=classpath*:com/webank/wedatasphere/dss/framework/appconn/dao/impl/*.xml +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.workflow.entity + +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.framework.appconn.dao + + +wds.linkis.server.mybatis.datasource.url=jdbc:mysql://127.0.0.1:3306/${databaseName}?characterEncoding=UTF-8 + +wds.linkis.server.mybatis.datasource.username= + +***REMOVED*** + +wds.linkis.gateway.ip=127.0.0.1 +wds.linkis.gateway.port=9001 + +wds.linkis.gateway.url=http://127.0.0.1:9001/ + +wds.linkis.mysql.is.encrypt=false \ No newline at end of file diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/resources/log4j.properties b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/log4j.properties new file mode 100644 index 000000000..ee8619595 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/log4j.properties @@ -0,0 +1,36 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +### set log levels ### + +log4j.rootCategory=INFO,console + +log4j.appender.console=org.apache.log4j.ConsoleAppender +log4j.appender.console.Threshold=INFO +log4j.appender.console.layout=org.apache.log4j.PatternLayout +#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n +log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n + + +log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender +log4j.appender.com.webank.bdp.ide.core.Threshold=INFO +log4j.additivity.com.webank.bdp.ide.core=false +log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout +log4j.appender.com.webank.bdp.ide.core.Append=true +log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log +log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n + +log4j.logger.org.springframework=INFO diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/resources/log4j2-engineconn.xml b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/log4j2-engineconn.xml new file mode 100644 index 000000000..5033fd642 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/log4j2-engineconn.xml @@ -0,0 +1,74 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/AppConnEngineConnPlugin.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/AppConnEngineConnPlugin.scala new file mode 100644 index 000000000..83f1b50de --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/AppConnEngineConnPlugin.scala @@ -0,0 +1,79 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn + +import java.util + +import com.webank.wedatasphere.linkis.manager.engineplugin.appconn.factory.AppConnEngineConnFactory +import com.webank.wedatasphere.linkis.manager.engineplugin.appconn.launch.AppConnProcessEngineConnLaunchBuilder +import com.webank.wedatasphere.linkis.manager.engineplugin.common.EngineConnPlugin +import com.webank.wedatasphere.linkis.manager.engineplugin.common.creation.EngineConnFactory +import com.webank.wedatasphere.linkis.manager.engineplugin.common.launch.EngineConnLaunchBuilder +import com.webank.wedatasphere.linkis.manager.engineplugin.common.resource.{EngineResourceFactory, GenericEngineResourceFactory} +import com.webank.wedatasphere.linkis.manager.label.entity.Label +import com.webank.wedatasphere.linkis.manager.label.entity.engine.EngineTypeLabel + +class AppConnEngineConnPlugin extends EngineConnPlugin { + + private val EP_CONTEXT_CONSTRUCTOR_LOCK = new Object() + + private var engineResourceFactory: EngineResourceFactory = _ + + private var engineLaunchBuilder: EngineConnLaunchBuilder = _ + + private var engineFactory: EngineConnFactory = _ + + private val defaultLabels: util.List[Label[_]] = new util.ArrayList[Label[_]]() + + + override def init(params: util.Map[String, Any]): Unit = { + val typeLabel =new EngineTypeLabel() + typeLabel.setEngineType("appconn") + typeLabel.setVersion("1.0.0") + this.defaultLabels.add(typeLabel) + } + + override def getEngineResourceFactory: EngineResourceFactory = { + if (null == engineResourceFactory) EP_CONTEXT_CONSTRUCTOR_LOCK.synchronized { + if (null == engineResourceFactory) { + engineResourceFactory = new GenericEngineResourceFactory + } + } + engineResourceFactory + } + + override def getEngineConnLaunchBuilder: EngineConnLaunchBuilder = { + if (null == engineLaunchBuilder) EP_CONTEXT_CONSTRUCTOR_LOCK.synchronized { + if (null == engineLaunchBuilder) { + engineLaunchBuilder = new AppConnProcessEngineConnLaunchBuilder + } + } + engineLaunchBuilder + } + + override def getEngineConnFactory: EngineConnFactory = { + if (null == engineFactory) EP_CONTEXT_CONSTRUCTOR_LOCK.synchronized { + if (null == engineFactory) { + engineFactory = new AppConnEngineConnFactory + } + } + engineFactory + } + + override def getDefaultLabels: util.List[Label[_]] = defaultLabels + +} diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/conf/AppConnEngineConnConfiguration.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/conf/AppConnEngineConnConfiguration.scala new file mode 100644 index 000000000..62611d8c6 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/conf/AppConnEngineConnConfiguration.scala @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.conf + +import com.webank.wedatasphere.linkis.common.conf.CommonVars + + +object AppConnEngineConnConfiguration { + + val GATEWAY_SPRING_APPLICATION = CommonVars("wds.linkis.gateway.spring.name", "dataworkcloud-gateway") + +} diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/exception/AppConnExecuteFailedException.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/exception/AppConnExecuteFailedException.scala new file mode 100644 index 000000000..30e92833b --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/exception/AppConnExecuteFailedException.scala @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.exception + +import com.webank.wedatasphere.linkis.common.exception.ErrorException + +case class AppConnExecuteFailedException (errCode:Int, desc:String) extends ErrorException(errCode, desc) diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/AbstractExecutionRequestRefContext.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/AbstractExecutionRequestRefContext.scala new file mode 100644 index 000000000..f485113d2 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/AbstractExecutionRequestRefContext.scala @@ -0,0 +1,115 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.executor + +import java.util + +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext +import com.webank.wedatasphere.dss.standard.app.development.listener.exception.AppConnExecutionErrorException +import com.webank.wedatasphere.linkis.common.io.resultset.{ResultSet, ResultSetReader, ResultSetWriter} +import com.webank.wedatasphere.linkis.common.io.{FsPath, MetaData, Record} +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.EngineExecutionContext +import com.webank.wedatasphere.linkis.manager.engineplugin.appconn.conf.AppConnEngineConnConfiguration +import com.webank.wedatasphere.linkis.protocol.UserWithCreator +import com.webank.wedatasphere.linkis.rpc.Sender +import com.webank.wedatasphere.linkis.storage.FSFactory +import com.webank.wedatasphere.linkis.storage.fs.FileSystem +import com.webank.wedatasphere.linkis.storage.resultset.{ResultSetFactory, ResultSetReader} + +import scala.collection.JavaConversions._ + +abstract class AbstractExecutionRequestRefContext(engineExecutorContext: EngineExecutionContext, userWithCreator: UserWithCreator) + extends ExecutionRequestRefContext with Logging { + + private var storePath: String = _ + + override def getRuntimeMap: util.Map[String, AnyRef] = engineExecutorContext.getProperties + + override def appendLog(log: String): Unit = engineExecutorContext.appendStdout(log) + + override def updateProgress(progress: Float): Unit = engineExecutorContext.pushProgress(progress, Array.empty) + + /** + * Get the operation user of this node. + */ + override def getUser: String = userWithCreator.user + + override def setStorePath(storePath: String): Unit = { + this.storePath = storePath + engineExecutorContext.setStorePath(storePath) + } + + override def getStorePath: String = storePath + + override def createTableResultSetWriter[M <: MetaData, R <: Record](): ResultSetWriter[M, R] = + createTableResultSetWriter(null) + + override def createTableResultSetWriter[M <: MetaData, R <: Record](resultSetAlias: String): ResultSetWriter[M, R] = + createResultSetWriter(ResultSetFactory.TABLE_TYPE, resultSetAlias) + + override def createTextResultSetWriter[M <: MetaData, R <: Record](): ResultSetWriter[M, R] = createTextResultSetWriter(null) + + override def createTextResultSetWriter[M <: MetaData, R <: Record](resultSetAlias: String): ResultSetWriter[M, R] = + createResultSetWriter(ResultSetFactory.TEXT_TYPE, resultSetAlias) + + override def createHTMLResultSetWriter[M <: MetaData, R <: Record](): ResultSetWriter[M, R] = createHTMLResultSetWriter(null) + + override def createHTMLResultSetWriter[M <: MetaData, R <: Record](resultSetAlias: String): ResultSetWriter[M, R] = + createResultSetWriter(ResultSetFactory.HTML_TYPE, resultSetAlias) + + override def createPictureResultSetWriter[M <: MetaData, R <: Record](): ResultSetWriter[M, R] = createPictureResultSetWriter(null) + + override def createPictureResultSetWriter[M <: MetaData, R <: Record](resultSetAlias: String): ResultSetWriter[M, R] = + createResultSetWriter(ResultSetFactory.PICTURE_TYPE, resultSetAlias) + + override def createResultSetWriter[M <: MetaData, R <: Record](resultSet: ResultSet[_ <: MetaData, _ <: Record], + resultSetAlias: String): ResultSetWriter[M, R] = + engineExecutorContext.createResultSetWriter(resultSet, resultSetAlias).asInstanceOf[ResultSetWriter[M, R]] + + override def getResultSetReader[M <: MetaData, R <: Record](fsPath: FsPath): ResultSetReader[M, R] = + ResultSetReader.getResultSetReader(fsPath.getSchemaPath).asInstanceOf[ResultSetReader[M, R]] + + private def createResultSetWriter[M <: MetaData, R <: Record](resultSetType: String, resultSetAlias: String): ResultSetWriter[M, R] = + engineExecutorContext.createResultSetWriter(resultSetType, resultSetAlias).asInstanceOf[ResultSetWriter[M, R]] + + override def sendResultSet(resultSetWriter: ResultSetWriter[_ <: MetaData, _ <: Record]): Unit = { + engineExecutorContext.sendResultSet(resultSetWriter) + } + + override def getGatewayUrl: String = { + val instances = Utils.tryThrow { + Sender.getInstances(AppConnEngineConnConfiguration.GATEWAY_SPRING_APPLICATION.getValue) + }{ t => new AppConnExecutionErrorException(75538, "获取gateway的url失败", t)} + if (instances.length == 0) throw new AppConnExecutionErrorException(75538, "获取gateway的url失败") + instances(0).getInstance + } + + override def fetchLinkisJobResultSetPaths(jobId: Long): Array[FsPath] = { + val task = fetchLinkisJob(jobId) + val resultSetLocation = task.getResultLocation + val user = task.getExecuteUser + FSFactory.getFsByProxyUser(new FsPath(resultSetLocation), user) match { + case fileSystem: FileSystem => + fileSystem.init(new util.HashMap[String, String]) + Utils.tryFinally { + import scala.collection.JavaConverters._ + fileSystem.listPathWithError(new FsPath(resultSetLocation)).getFsPaths.asScala.toArray[FsPath] + }(Utils.tryQuietly(fileSystem.close())) + } + } +} \ No newline at end of file diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/AppConnEngineConnExecutor.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/AppConnEngineConnExecutor.scala new file mode 100644 index 000000000..7705cf7f4 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/AppConnEngineConnExecutor.scala @@ -0,0 +1,188 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.executor + +import java.util +import java.util.Map + +import com.webank.wedatasphere.dss.appconn.core.AppConn +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager +import com.webank.wedatasphere.dss.common.label.{EnvDSSLabel, LabelKeyConvertor} +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.dss.standard.app.development.listener.common.{AsyncExecutionRequestRef, CompletedExecutionResponseRef} +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.sso.Workspace +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance +import com.webank.wedatasphere.dss.standard.common.entity.ref.{AsyncResponseRef, DefaultRefFactory, ResponseRef} +import com.webank.wedatasphere.linkis.common.utils.{OverloadUtils, Utils} +import com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.{ComputationExecutor, EngineExecutionContext} +import com.webank.wedatasphere.linkis.engineconn.launch.EngineConnServer +import com.webank.wedatasphere.linkis.governance.common.utils.GovernanceConstant +import com.webank.wedatasphere.linkis.manager.common.entity.resource.{CommonNodeResource, LoadResource, NodeResource} +import com.webank.wedatasphere.linkis.manager.engineplugin.appconn.exception.AppConnExecuteFailedException +import com.webank.wedatasphere.linkis.manager.engineplugin.appconn.executor.AppConnEngineConnExecutor._ +import com.webank.wedatasphere.linkis.manager.engineplugin.common.conf.EngineConnPluginConf +import com.webank.wedatasphere.linkis.manager.label.entity.Label +import com.webank.wedatasphere.linkis.manager.label.entity.cluster.ClusterLabel +import com.webank.wedatasphere.linkis.manager.label.entity.engine.EngineTypeLabel +import com.webank.wedatasphere.linkis.protocol.UserWithCreator +import com.webank.wedatasphere.linkis.protocol.engine.JobProgressInfo +import com.webank.wedatasphere.linkis.scheduler.executer.{ErrorExecuteResponse, ExecuteResponse, SuccessExecuteResponse} +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper +import org.apache.commons.lang.StringUtils + +import scala.beans.BeanProperty + +class AppConnEngineConnExecutor(val id: Int) extends ComputationExecutor { + + @BeanProperty + var userWithCreator: UserWithCreator = _ + + private val executorLabels: util.List[Label[_]] = new util.ArrayList[Label[_]](2) + + private val refFactory = new DefaultRefFactory + + override def executeLine(engineExecutorContext: EngineExecutionContext, code: String): ExecuteResponse = { + info("got code:[" + code + "]") + info("got params:[" + BDPJettyServerHelper.gson.toJson(engineExecutorContext.getProperties) + "]") + val source = engineExecutorContext.getProperties.get(GovernanceConstant.TASK_SOURCE_MAP_KEY) match { + case map: util.Map[String, Object] => map + case _ => return ErrorExecuteResponse("Cannot find source.", null) + } + def getValue(map: util.Map[String, AnyRef], key: String): String = map.get(key) match { + case string: String => string + case anyRef: AnyRef => BDPJettyServerHelper.gson.toJson(anyRef) + case _ => null + } + val requestRef = refFactory.newRef(classOf[ExecutionRequestRef], getClass.getClassLoader, "com.webank.wedatasphere") match { + case ref: AsyncExecutionRequestRef => + ref.setProjectName(getValue(source, PROJECT_NAME_STR)) + ref.setOrchestratorName(getValue(source, FLOW_NAME_STR)) + ref.setJobContent(BDPJettyServerHelper.gson.fromJson(code, classOf[util.HashMap[String, AnyRef]])) + ref.setName(getValue(source, NODE_NAME_STR)) + engineExecutorContext.getLabels.find(_.isInstanceOf[EngineTypeLabel]).foreach { case engineTypeLabel: EngineTypeLabel => + ref.setType(engineTypeLabel.getEngineType) + } + ref.setExecutionRequestRefContext(new ExecutionRequestRefContextImpl(engineExecutorContext, userWithCreator)) + ref + case ref: ExecutionRequestRef => ref + } + val workspace = BDPJettyServerHelper.gson.fromJson(getValue(engineExecutorContext.getProperties, WORKSPACE_NAME_STR), classOf[Workspace]) + requestRef.setWorkspace(workspace) + val appConnName = getAppConnName(getValue(engineExecutorContext.getProperties, NODE_TYPE)) + val appConn = AppConnManager.getAppConnManager.getAppConn(appConnName) + //val labels = getDSSLabels(engineExecutorContext.getLabels) + var labels = engineExecutorContext.getProperties.get("labels").toString + getAppInstanceByLabels(labels, appConn) match { + case Some(appInstance) => + val developmentIntegrationStandard = appConn.asInstanceOf[OnlyDevelopmentAppConn].getOrCreateDevelopmentStandard + val refExecutionService =developmentIntegrationStandard.getRefExecutionService(appInstance) + val refExecutionOperation = refExecutionService.getRefExecutionOperation + Utils.tryCatch(refExecutionOperation.execute(requestRef) match { + case asyncResponseRef: AsyncResponseRef => + asyncResponseRef.waitForCompleted() + createExecuteResponse(asyncResponseRef.getResponse, appConnName) + case responseRef: ResponseRef => + createExecuteResponse(responseRef, appConnName) + })(ErrorExecuteResponse("Failed to execute appconn", _)) + case None => + throw AppConnExecuteFailedException(510000, "Cannot Find AppInstance by labels " + labels) + } + } + + private def getDSSLabels(labels: Array[Label[_]]): String = + labels.find(_.isInstanceOf[ClusterLabel]).map {case clusterLabel: ClusterLabel => clusterLabel.getClusterName } + .getOrElse(throw AppConnExecuteFailedException(510000, "Cannot Find AppInstance by labels " + labels.toList)) + + private def createExecuteResponse(responseRef: ResponseRef, appConnName: String): ExecuteResponse = + if(responseRef.isSucceed) SuccessExecuteResponse() + else { + val exception = responseRef match { + case response: CompletedExecutionResponseRef => response.getException + case _ => null + } + error(s"$appConnName execute failed, failed reason is ${responseRef.getErrorMsg}.", exception) + ErrorExecuteResponse(responseRef.getErrorMsg, exception) + } + + private def getAppConnName(nodeType: String) = { + StringUtils.split(nodeType,".")(0) + } + + private def getAppInstanceByLabels(labels: String, appConn: AppConn): Option[AppInstance] = { + var labelStr = labels + if (labels.contains(LabelKeyConvertor.ROUTE_LABEL_KEY)) { + val labelMap = DSSCommonUtils.COMMON_GSON.fromJson(labels, classOf[util.Map[_, _]]) + labelStr = labelMap.get(LabelKeyConvertor.ROUTE_LABEL_KEY).asInstanceOf[String] + } + val appInstanceList = appConn.getAppDesc.getAppInstancesByLabels(util.Arrays.asList(new EnvDSSLabel(labelStr))); + if (appInstanceList != null && appInstanceList.size() > 0) { + return Some(appInstanceList.get(0)) + } + None + } + + override def executeCompletely(engineExecutorContext: EngineExecutionContext, code: String, completedLine: String): ExecuteResponse = null + + override def progress(): Float = 0 + + override def getProgressInfo: Array[JobProgressInfo] = Array.empty + + override def supportCallBackLogs(): Boolean = false + + override def getExecutorLabels(): util.List[Label[_]] = executorLabels + + override def setExecutorLabels(labels: util.List[Label[_]]): Unit = { + if (null != labels && !labels.isEmpty) { + executorLabels.clear() + executorLabels.addAll(labels) + } + } + + override def requestExpectedResource(expectedResource: NodeResource): NodeResource = null + + override def getCurrentNodeResource(): NodeResource = { + val properties = EngineConnServer.getEngineCreationContext.getOptions + if (properties.containsKey(EngineConnPluginConf.JAVA_ENGINE_REQUEST_MEMORY.key)) { + val settingClientMemory = properties.get(EngineConnPluginConf.JAVA_ENGINE_REQUEST_MEMORY.key) + if (!settingClientMemory.toLowerCase().endsWith("g")) { + properties.put(EngineConnPluginConf.JAVA_ENGINE_REQUEST_MEMORY.key, settingClientMemory + "g") + } + } + val resource = new CommonNodeResource + val usedResource = new LoadResource(OverloadUtils.getProcessMaxMemory, 1) + resource.setUsedResource(usedResource) + resource + } + + override def getId(): String = "AppConnEngineExecutor_" + id + +} +object AppConnEngineConnExecutor { + + private val WORKSPACE_NAME_STR = "workspace" + + private val PROJECT_NAME_STR = "projectName" + + private val FLOW_NAME_STR = "flowName" + + private val NODE_NAME_STR = "nodeName" + + private val NODE_TYPE = "nodeType" + +} \ No newline at end of file diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/CommonExecutionRequestRef.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/CommonExecutionRequestRef.scala new file mode 100644 index 000000000..f3a4d049d --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/CommonExecutionRequestRef.scala @@ -0,0 +1,90 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.executor + +import java.util + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionRequestRef +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext +import com.webank.wedatasphere.dss.standard.app.sso.Workspace + +class CommonExecutionRequestRef extends AsyncExecutionRequestRef { + + private var workspace: Workspace = _ + private var projectId: Long = _ + private var projectName: String = _ + private var orchestratorName: String = _ + private var orchestratorVersion: String = _ + private var orchestratorId: Long = _ + private var jobContent: util.Map[String, AnyRef] = _ + private var executionRequestRefContext: ExecutionRequestRefContext = _ + + private val parameters: util.Map[String, AnyRef] = new util.HashMap[String, AnyRef]() + private var name: String = _ + private var refType: String = _ + + + override def setWorkspace(workspace: Workspace): Unit = { + this.workspace = workspace + } + + override def getWorkspace: Workspace = workspace + + /** + * Gets the ID of the project to which the node belongs + */ + override def getProjectId: Long = projectId + def setProjectId(projectId: Long): Unit = this.projectId = projectId + + /** + * Get the name of the project to which node belongs + */ + override def getProjectName: String = projectName + override def setProjectName(projectName: String): Unit = this.projectName = projectName + + override def getOrchestratorName: String = orchestratorName + override def setOrchestratorName(orchestratorName: String): Unit = this.orchestratorName = orchestratorName + + override def getOrchestratorVersion: String = orchestratorVersion + def setOrchestratorVersion(orchestratorVersion: String): Unit = this.orchestratorVersion = orchestratorVersion + + override def getOrchestratorId: Long = orchestratorId + def setOrchestratorId(orchestratorId: Long): Unit = this.orchestratorId = orchestratorId + + /** + * Get the execution content of the node. The execution content is in the form of map + */ + override def getJobContent: util.Map[String, AnyRef] = jobContent + override def setJobContent(jobContent: util.Map[String, AnyRef]): Unit = this.jobContent = jobContent + + override def getParameter(key: String): AnyRef = parameters.get(key) + + override def setParameter(key: String, value: AnyRef): Unit = parameters.put(key, value) + + override def getParameters: util.Map[String, AnyRef] = parameters + + override def getName: String = name + override def setName(name: String): Unit = this.name = name + + override def getType: String = refType + override def setType(`type`: String): Unit = this.refType = `type` + + override def getExecutionRequestRefContext: ExecutionRequestRefContext = executionRequestRefContext + + override def setExecutionRequestRefContext(executionRequestRefContext: ExecutionRequestRefContext): Unit = + this.executionRequestRefContext = executionRequestRefContext +} diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/ExecutionRequestRefContextImpl.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/ExecutionRequestRefContextImpl.scala new file mode 100644 index 000000000..779d5b229 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/executor/ExecutionRequestRefContextImpl.scala @@ -0,0 +1,87 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.executor + +import java.util + +import com.webank.wedatasphere.dss.standard.app.development.listener.conf.RefExecutionConfiguration +import com.webank.wedatasphere.dss.standard.app.development.listener.core.LinkisJob +import com.webank.wedatasphere.dss.standard.app.development.listener.exception.AppConnExecutionErrorException +import com.webank.wedatasphere.linkis.common.exception.ErrorException +import com.webank.wedatasphere.linkis.common.utils.Utils +import com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.EngineExecutionContext +import com.webank.wedatasphere.linkis.governance.common.entity.job.JobRequestWithDetail +import com.webank.wedatasphere.linkis.governance.common.protocol.job.{JobReqQuery, JobRespProtocol} +import com.webank.wedatasphere.linkis.manager.label.entity.Label +import com.webank.wedatasphere.linkis.protocol.UserWithCreator +import com.webank.wedatasphere.linkis.rpc.Sender + +import scala.collection.JavaConversions._ + +/** + * Only jobhistory of linkis1.0 is applicable. + */ +class ExecutionRequestRefContextImpl(engineExecutorContext: EngineExecutionContext, userWithCreator: UserWithCreator) + extends AbstractExecutionRequestRefContext(engineExecutorContext, userWithCreator) { + + private val jobHistorySender:Sender = Sender.getSender(RefExecutionConfiguration.JOB_HISTORY_APPLICATION_NAME.getValue) + + override def fetchLinkisJob(jobId: Long): LinkisJob = { + val jobRequest:JobRequestWithDetail= new JobRequestWithDetail() + jobRequest.setId(jobId) + jobRequest.setSource(null) + val requestQueryTask = JobReqQuery(jobRequest) + val linkisJob:JobRequestWithDetail = Utils.tryThrow( + jobHistorySender.ask(requestQueryTask) match { + case responsePersist: JobRespProtocol => + val status = responsePersist.getStatus + if (status != 0){ + error(s"Fetch linkisJob from jobHistory failed, errorMsg: ${responsePersist.getMsg}.") + throw new AppConnExecutionErrorException(95541, s"Fetch linkisJob from jobHistory failed, errorMsg: ${responsePersist.getMsg}.") + } else { + val data = responsePersist.getData + data.get("jobHistoryList") match { + case tasks: util.List[JobRequestWithDetail] => + if (tasks.size() > 0) tasks.get(0) else throw new AppConnExecutionErrorException(95542, s"query from jobhistory not a correct List type taskId is $jobId") + + + case _ => throw new AppConnExecutionErrorException(95541, s"query from jobhistory not a correct List type taskId is $jobId") + } + } + + + case r => + error(s"Fetch linkisJob from jobHistory incorrectly, response is $r.") + throw new AppConnExecutionErrorException(95541, s"Fetch linkisJob from jobHistory incorrectly, response is $r.") + } + ){ + case errorException: ErrorException => errorException + case e: Exception => + new AppConnExecutionErrorException(95541, s"query taskId $jobId error.", e) + } + new LinkisJob { + override def getResultLocation: String = linkisJob.getSubJobDetailList.find(subJob =>{null != subJob.getResultLocation}).get.getResultLocation + override def getSubmitUser: String = linkisJob.getSubmitUser + override def getExecuteUser: String = linkisJob.getSubmitUser + override def getStatus: String = linkisJob.getStatus + override def getSource: util.Map[String, String] = linkisJob.getSource + override def getParams: util.Map[String, AnyRef] = linkisJob.getParams + override def getLabels: util.List[Label[_]] = linkisJob.getLabels + } + } + +} diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/factory/AppConnEngineConnFactory.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/factory/AppConnEngineConnFactory.scala new file mode 100644 index 000000000..d00b01376 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/factory/AppConnEngineConnFactory.scala @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.factory + +import java.io.File + +import com.webank.wedatasphere.dss.appconn.loader.utils.AppConnUtils +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.engineconn.common.conf.EngineConnConf +import com.webank.wedatasphere.linkis.engineconn.common.creation.EngineCreationContext +import com.webank.wedatasphere.linkis.engineconn.common.engineconn.EngineConn +import com.webank.wedatasphere.linkis.engineconn.computation.executor.creation.ComputationSingleExecutorEngineConnFactory +import com.webank.wedatasphere.linkis.engineconn.executor.entity.LabelExecutor +import com.webank.wedatasphere.linkis.governance.common.exception.engineconn.EngineConnExecutorErrorCode +import com.webank.wedatasphere.linkis.manager.engineplugin.appconn.executor.AppConnEngineConnExecutor +import com.webank.wedatasphere.linkis.manager.engineplugin.common.exception.EngineConnPluginErrorException +import com.webank.wedatasphere.linkis.manager.label.entity.engine.EngineType.EngineType +import com.webank.wedatasphere.linkis.manager.label.entity.engine.RunType.RunType +import com.webank.wedatasphere.linkis.manager.label.entity.engine.{EngineType, RunType} + +class AppConnEngineConnFactory extends ComputationSingleExecutorEngineConnFactory { + + private val appConnHomePath = new File(EngineConnConf.getWorkHome, AppConnUtils.APPCONN_DIR_NAME) + if(!appConnHomePath.exists() && !appConnHomePath.mkdir()) + throw new EngineConnPluginErrorException(EngineConnExecutorErrorCode.INIT_EXECUTOR_FAILED, + s"Cannot mkdir ${appConnHomePath.getPath}, please make sure the permission is ok.") + DataWorkCloudApplication.setProperty(AppConnUtils.APPCONN_HOME_PATH.key, appConnHomePath.getPath) + warn(s"Set ${AppConnUtils.APPCONN_HOME_PATH.key}=${appConnHomePath.getPath}.") + + override protected def newExecutor(id: Int, + engineCreationContext: EngineCreationContext, + engineConn: EngineConn): LabelExecutor = + new AppConnEngineConnExecutor(id) + + + override protected def getEngineConnType: EngineType = EngineType.APPCONN + + override protected def getRunType: RunType = RunType.APPCONN + +} \ No newline at end of file diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/launch/AppConnProcessEngineConnLaunchBuilder.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/launch/AppConnProcessEngineConnLaunchBuilder.scala new file mode 100644 index 000000000..dd42a8a11 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/launch/AppConnProcessEngineConnLaunchBuilder.scala @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.launch + +import java.util + +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager +import com.webank.wedatasphere.dss.appconn.manager.impl.AbstractAppConnManager +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnInfoService +import com.webank.wedatasphere.linkis.common.utils.Utils +import com.webank.wedatasphere.linkis.manager.common.protocol.bml.BmlResource +import com.webank.wedatasphere.linkis.manager.common.protocol.bml.BmlResource.BmlResourceVisibility +import com.webank.wedatasphere.linkis.manager.engineplugin.common.launch.entity.EngineConnBuildRequest +import com.webank.wedatasphere.linkis.manager.engineplugin.common.launch.process.JavaProcessEngineConnLaunchBuilder + +import scala.collection.convert.wrapAsScala._ +import scala.collection.convert.wrapAsJava._ + +class AppConnProcessEngineConnLaunchBuilder extends JavaProcessEngineConnLaunchBuilder{ + + override protected def getBmlResources(implicit engineConnBuildRequest: EngineConnBuildRequest): util.List[BmlResource] = { + super.getBmlResources + } + +} diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/listener/AppConnEngineExecutionLogListener.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/listener/AppConnEngineExecutionLogListener.scala new file mode 100644 index 000000000..5e3f92736 --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/listener/AppConnEngineExecutionLogListener.scala @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.listener + +import com.webank.wedatasphere.dss.standard.app.development.listener.ExecutionLogListener +import com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.EngineExecutionContext + +class AppConnEngineExecutionLogListener(val engineExecutionContext: EngineExecutionContext) extends ExecutionLogListener { + + override def onInfo(log: String): Unit = { + engineExecutionContext.info(log) + } + + override def onERROR(log: String): Unit = { + engineExecutionContext.error(log) + } + + override def onWarn(log: String): Unit = { + engineExecutionContext.warn(log) + } + + override def onSystemInfo(log: String): Unit = { + engineExecutionContext.info(log) + } + + override def onSystemError(log: String): Unit = { + engineExecutionContext.error(log) + } + + override def onSystemWarn(log: String): Unit = { + engineExecutionContext.warn(log) + } + +} diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/listener/AppConnEngineExecutionResultListener.scala b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/listener/AppConnEngineExecutionResultListener.scala new file mode 100644 index 000000000..ff4169b1d --- /dev/null +++ b/dss-appconn/linkis-appconn-engineplugin/src/main/scala/com/webank/wedatasphere/linkis/manager/engineplugin/appconn/listener/AppConnEngineExecutionResultListener.scala @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.manager.engineplugin.appconn.listener + +import com.webank.wedatasphere.dss.standard.app.development.listener.ExecutionResultListener +import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetWriter +import com.webank.wedatasphere.linkis.common.io.{MetaData, Record} +import com.webank.wedatasphere.linkis.engineconn.computation.executor.execute.EngineExecutionContext + +class AppConnEngineExecutionResultListener(val engineExecutionContext: EngineExecutionContext) extends ExecutionResultListener { + + private var resultSetWriter: ResultSetWriter[_ <: MetaData, _ <: Record] = _ + + override def setResultSetType(resultSetType: String): Unit = { + resultSetWriter = engineExecutionContext.createResultSetWriter(resultSetType) + } + + override def onResultMetaData(metaData: MetaData): Unit = { + if(resultSetWriter == null){ + resultSetWriter = engineExecutionContext.createDefaultResultSetWriter() + } + resultSetWriter.addMetaData(metaData) + } + + override def onResultSetRecord(record: Record): Unit = { + if(resultSetWriter == null){ + resultSetWriter = engineExecutionContext.createDefaultResultSetWriter() + } + resultSetWriter.addRecord(record) + } + +} diff --git a/dss-appconn/pom.xml b/dss-appconn/pom.xml new file mode 100644 index 000000000..2821a3662 --- /dev/null +++ b/dss-appconn/pom.xml @@ -0,0 +1,47 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../pom.xml + + 4.0.0 + + dss-appconn + pom + + + dss-appconn-core + dss-appconn-loader + dss-appconn-manager + dss-scheduler-appconn + linkis-appconn-engineplugin + appconns/dss-datachecker-appconn + appconns/dss-eventchecker-appconn + appconns/dss-orchestrator-framework-appconn + appconns/dss-schedulis-appconn + appconns/dss-workflow-appconn + appconns/dss-sendemail-appconn + appconns/dss-visualis-appconn + + + \ No newline at end of file diff --git a/dss-appjoint-auth/pom.xml b/dss-appjoint-auth/pom.xml deleted file mode 100644 index 3a368b866..000000000 --- a/dss-appjoint-auth/pom.xml +++ /dev/null @@ -1,75 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-appjoint-auth - - - com.webank.wedatasphere.linkis - linkis-gateway-httpclient-support - ${linkis.version} - - - com.webank.wedatasphere.linkis - linkis-common - ${linkis.version} - - - javax.servlet - javax.servlet-api - 3.1.0 - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-compiler-plugin - 3.3 - - 1.8 - 1.8 - UTF-8 - - - - - - \ No newline at end of file diff --git a/dss-appjoint-auth/src/main/java/com/webank/wedatasphere/dss/appjoint/auth/AppJointAuth.java b/dss-appjoint-auth/src/main/java/com/webank/wedatasphere/dss/appjoint/auth/AppJointAuth.java deleted file mode 100644 index aa03f8dc6..000000000 --- a/dss-appjoint-auth/src/main/java/com/webank/wedatasphere/dss/appjoint/auth/AppJointAuth.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.auth; - -import com.webank.wedatasphere.dss.appjoint.auth.impl.AppJointAuthImpl$; - -import javax.servlet.http.HttpServletRequest; -import java.io.Closeable; - -/** - * Created by enjoyyin on 2019/11/6. - */ -public interface AppJointAuth extends Closeable { - - boolean isDssRequest(HttpServletRequest request); - - RedirectMsg getRedirectMsg(HttpServletRequest request); - - static AppJointAuth getAppJointAuth() { - return AppJointAuthImpl$.MODULE$.apply(); - } - -} diff --git a/dss-appjoint-auth/src/main/java/com/webank/wedatasphere/dss/appjoint/auth/RedirectMsg.java b/dss-appjoint-auth/src/main/java/com/webank/wedatasphere/dss/appjoint/auth/RedirectMsg.java deleted file mode 100644 index b4875a196..000000000 --- a/dss-appjoint-auth/src/main/java/com/webank/wedatasphere/dss/appjoint/auth/RedirectMsg.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.auth; - -/** - * Created by enjoyyin on 2019/11/7. - */ -public interface RedirectMsg { - - String getRedirectUrl(); - - void setRedirectUrl(String redirectUrl); - - String getUser(); - - void setUser(String user); - -} diff --git a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/AppJointAuthImpl.scala b/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/AppJointAuthImpl.scala deleted file mode 100644 index 15dd7e307..000000000 --- a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/AppJointAuthImpl.scala +++ /dev/null @@ -1,99 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.auth.impl - -import java.net.URI -import java.util - -import com.webank.wedatasphere.dss.appjoint.auth.{AppJointAuth, RedirectMsg} -import com.webank.wedatasphere.linkis.common.utils.Logging -import com.webank.wedatasphere.linkis.httpclient.dws.DWSHttpClient -import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfigBuilder -import javax.servlet.http.HttpServletRequest -import org.apache.commons.io.IOUtils -import org.apache.http.impl.cookie.BasicClientCookie - -import scala.collection.JavaConversions._ - -/** - * Created by enjoyyin on 2019/11/6. - */ -class AppJointAuthImpl private() extends AppJointAuth with Logging { - - private val dwsHttpClients = new util.HashMap[String, DWSHttpClient] - - private def getBaseUrl(dssUrl: String): String = { - val uri = new URI(dssUrl) - val dssPort = if(uri.getPort != -1) uri.getPort else 80 - uri.getScheme + "://" + uri.getHost + ":" + dssPort - } - - protected def getDWSClient(dssUrl: String): DWSHttpClient = { - val baseUrl = getBaseUrl(dssUrl) - if(!dwsHttpClients.containsKey(baseUrl)) baseUrl.intern synchronized { - if(!dwsHttpClients.containsKey(baseUrl)) { - info("create a new DSSClient for url " + baseUrl) - val clientConfig = DWSClientConfigBuilder.newBuilder().setDWSVersion(AppJointAuthImpl.dssVersion) - .addUJESServerUrl(baseUrl).connectionTimeout(30000).discoveryEnabled(false) - .maxConnectionSize(AppJointAuthImpl.maxConnection).readTimeout(30000).build() - val dwsHttpClient = new DWSHttpClient(clientConfig, "DSS-UserInfo-Ask-Client") - dwsHttpClients.put(baseUrl, dwsHttpClient) - } - } - dwsHttpClients.get(baseUrl) - } - - override def isDssRequest(request: HttpServletRequest): Boolean = request.getParameterMap.containsKey(AppJointAuthImpl.DSS_URL_KEY) - - override def getRedirectMsg(request: HttpServletRequest): RedirectMsg = { - val dssUrl = request.getParameter(AppJointAuthImpl.DSS_URL_KEY) - val dwsHttpClient = getDWSClient(dssUrl) - val userInfoAction = new UserInfoAction - val cookies = request.getParameter(AppJointAuthImpl.COOKIES_KEY) - cookies.split(";").foreach { cookie => - val index = cookie.indexOf("=") - val key = cookie.substring(0, index).trim - val value = cookie.substring(index + 1).trim - userInfoAction.addCookie(new BasicClientCookie(key, value)) - } - val redirectMsg = new RedirectMsgImpl - redirectMsg.setRedirectUrl(request.getParameter(AppJointAuthImpl.REDIRECT_KEY)) - dwsHttpClient.execute(userInfoAction) match { - case userInfoResult: UserInfoResult => - redirectMsg.setUser(userInfoResult.getUserName) - } - redirectMsg - } - - override def close(): Unit = dwsHttpClients.values().foreach(IOUtils.closeQuietly) -} - -object AppJointAuthImpl { - private val appJointAuth = new AppJointAuthImpl - private var dssVersion = "v1" - private var maxConnection = 5 - - private val REDIRECT_KEY = "redirect" - private val DSS_URL_KEY = "dssurl" - private val COOKIES_KEY = "cookies" - - def apply: AppJointAuthImpl = appJointAuth - - def setDSSVersion(version: String): Unit = this.dssVersion = version - def setMaxConnection(maxConnection: Int): Unit = this.maxConnection = maxConnection -} \ No newline at end of file diff --git a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/RedirectMsgImpl.scala b/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/RedirectMsgImpl.scala deleted file mode 100644 index ef2c8da74..000000000 --- a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/RedirectMsgImpl.scala +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.auth.impl - -import com.webank.wedatasphere.dss.appjoint.auth.RedirectMsg - -/** - * Created by enjoyyin on 2019/11/7. - */ -class RedirectMsgImpl extends RedirectMsg { - - private var redirectUrl: String = _ - private var user: String = _ - - override def getRedirectUrl: String = redirectUrl - - override def setRedirectUrl(redirectUrl: String): Unit = this.redirectUrl = redirectUrl - - override def getUser: String = user - - override def setUser(user: String): Unit = this.user = user -} diff --git a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/UserInfoAction.scala b/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/UserInfoAction.scala deleted file mode 100644 index 6d454b2ff..000000000 --- a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/UserInfoAction.scala +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.auth.impl - -import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction -import com.webank.wedatasphere.linkis.httpclient.request.GetAction - -/** - * Created by enjoyyin on 2019/11/7. - */ -class UserInfoAction extends GetAction with DWSHttpAction { - override def suffixURLs: Array[String] = Array("user", "userInfo") -} diff --git a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/UserInfoResult.scala b/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/UserInfoResult.scala deleted file mode 100644 index f0280b8cf..000000000 --- a/dss-appjoint-auth/src/main/scala/com/webank/wedatasphere/dss/appjoint/auth/impl/UserInfoResult.scala +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.auth.impl - -import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult -import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult - -import scala.beans.BeanProperty - -/** - * Created by enjoyyin on 2019/11/7. - */ -@DWSHttpMessageResult("/api/rest_j/v\\d+/user/userInfo") -class UserInfoResult extends DWSResult { - - @BeanProperty var userName: String = _ - -} diff --git a/dss-appjoint-core/pom.xml b/dss-appjoint-core/pom.xml deleted file mode 100644 index 7140b3365..000000000 --- a/dss-appjoint-core/pom.xml +++ /dev/null @@ -1,93 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-appjoint-core - - - - com.webank.wedatasphere.linkis - linkis-protocol - ${linkis.version} - - - com.webank.wedatasphere.linkis - linkis-storage - ${linkis.version} - - - - - - - - - com.webank.wedatasphere.dss - dss-common - ${dss.version} - - - - com.webank.wedatasphere.linkis - linkis-httpclient - ${linkis.version} - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-compiler-plugin - 3.3 - - 1.8 - 1.8 - UTF-8 - - - - - - - - - - \ No newline at end of file diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/AppJoint.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/AppJoint.java deleted file mode 100644 index 10d20f11c..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/AppJoint.java +++ /dev/null @@ -1,73 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution; -import com.webank.wedatasphere.dss.appjoint.service.NodeService; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import com.webank.wedatasphere.dss.appjoint.service.SecurityService; - -import java.util.Map; - -/** - * Created by enjoyyin on 2019/10/10. - */ -public interface AppJoint { - - /** - * 每一个AppJoint都需要有一个自己的标识名
- * 这个标识名是唯一的，如qualitis子系统，可以将名字设置为qualitis; visualis子系统，可以设置为visualis - */ - String getAppJointName(); - - void init(String baseUrl, Map params) throws AppJointErrorException; - - /** - * 如果不存在，直接返回null即可 - * @return - */ - default SecurityService getSecurityService() { - return null; - } - - /** - * 如果不存在，直接返回null即可 - * @return - */ - default ProjectService getProjectService() { - return null; - } - - /** - * 如果不存在，直接返回null即可 - * @return - */ - default NodeService getNodeService() { - return null; - } - - /** - * 如果不存在，直接返回null即可 - * @return - */ - default NodeExecution getNodeExecution() { - return null; - } - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/exception/AppJointErrorException.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/exception/AppJointErrorException.java deleted file mode 100644 index 86d074928..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/exception/AppJointErrorException.java +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.exception; - -import com.webank.wedatasphere.linkis.common.exception.ErrorException; - -/** - * Created by enjoyyin on 2019/11/5. - */ -public class AppJointErrorException extends ErrorException { - - public AppJointErrorException(int errCode, String desc) { - super(errCode, desc); - } - - public AppJointErrorException(int errCode, String desc, Throwable cause) { - super(errCode, desc); - initCause(cause); - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/NodeExecution.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/NodeExecution.java deleted file mode 100644 index 6cd179492..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/NodeExecution.java +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.execution.core.NodeContext; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrl; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - - -/** - * created by enjoyyin on 2019/9/25 - * Description: - * NodeExecution - */ -public interface NodeExecution extends AppJointUrl { - - void init(java.util.Map params) throws AppJointErrorException; - - /** - * 表示任务能否提交到该AppJoint去执行 - * @param appJointNode AppJointNode - * @return true is ok while false is not - */ - boolean canExecute(AppJointNode appJointNode, NodeContext context, Session session); - - /** - * 相应的appJoint提交到外部系统执行 - * @param appJointNode 就是工作流节点 - * @param context 运行时参数的context - * @param session session是为了解决将任务提交给第三方系统时解决鉴权等问题 - */ - NodeExecutionResponse execute(AppJointNode appJointNode, NodeContext context, Session session) throws AppJointErrorException; - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionListener.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionListener.java deleted file mode 100644 index 4502660f1..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionListener.java +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.async; - -import com.webank.wedatasphere.dss.appjoint.execution.common.AsyncNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.common.CompletedNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.execution.core.NodeContext; - -/** - * created by enjoyyin on 2019/10/8 - * Description: - */ -public interface NodeExecutionListener { - - void beforeSubmit(AppJointNode appJointNode, NodeContext nodeContext); - - void afterSubmit(AppJointNode appJointNode, NodeContext nodeContext, NodeExecutionAction action); - - void afterAsyncNodeExecutionResponse(AsyncNodeExecutionResponse response); - - void afterCompletedNodeExecutionResponse(AppJointNode appJointNode, NodeContext nodeContext, NodeExecutionAction action, - CompletedNodeExecutionResponse response); -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionLogListener.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionLogListener.java deleted file mode 100644 index 7959042ed..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionLogListener.java +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.async; - -/** - * created by enjoyyin on 2019/9/30 - * Description: - */ -public interface NodeExecutionLogListener { - public void onLogUpdated(String log); -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionProgressListener.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionProgressListener.java deleted file mode 100644 index b1955a617..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionProgressListener.java +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.async; - -/** - * created by enjoyyin on 2019/9/30 - * Description: - */ -public interface NodeExecutionProgressListener { - public void onProgressUpdated(double progress); -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionResponseListener.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionResponseListener.java deleted file mode 100644 index 8cc32bdb8..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionResponseListener.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.async; - -import com.webank.wedatasphere.dss.appjoint.execution.common.CompletedNodeExecutionResponse; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public interface NodeExecutionResponseListener { - - void onNodeExecutionCompleted(CompletedNodeExecutionResponse response); - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionStatusListener.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionStatusListener.java deleted file mode 100644 index a606638f3..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/async/NodeExecutionStatusListener.java +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.async; - -/** - * created by enjoyyin on 2019/9/30 - * Description: - */ -public interface NodeExecutionStatusListener { - public void onStatusUpdated(String status); -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/AbstractNodeExecutionAction.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/AbstractNodeExecutionAction.java deleted file mode 100644 index 495f77397..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/AbstractNodeExecutionAction.java +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -/** - * created by enjoyyin on 2019/9/26 - * Description: - */ -public abstract class AbstractNodeExecutionAction implements NodeExecutionAction { - - public String id; - - public String getId() { - return id; - } - - public void setId(String id) { - this.id = id; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/AsyncNodeExecutionResponse.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/AsyncNodeExecutionResponse.java deleted file mode 100644 index a0fb502fd..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/AsyncNodeExecutionResponse.java +++ /dev/null @@ -1,116 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -import com.webank.wedatasphere.dss.appjoint.execution.async.NodeExecutionResponseListener; -import com.webank.wedatasphere.dss.appjoint.execution.core.LongTermNodeExecution; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.execution.core.NodeContext; - -import java.util.ArrayList; -import java.util.List; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public class AsyncNodeExecutionResponse implements NodeExecutionResponse { - - private NodeExecutionAction action; - private LongTermNodeExecution nodeExecution; - private AppJointNode appJointNode; - private NodeContext nodeContext; - private boolean isCompleted = false; - private long startTime = System.currentTimeMillis(); - private long askStatePeriod = 1000; - private long maxLoopTime = -1; - - /** - * 通过监听的方式，任务一旦完成，我们就通知上层 - */ - private List listener = new ArrayList<>(); - - - public NodeExecutionAction getAction() { - return action; - } - - public void setAction(NodeExecutionAction action) { - this.action = action; - } - - public void addListener(NodeExecutionResponseListener responseListener){ - this.listener.add(responseListener); - } - - public List getListeners(){ - return this.listener; - } - - public AppJointNode getAppJointNode() { - return appJointNode; - } - - public void setAppJointNode(AppJointNode appJointNode) { - this.appJointNode = appJointNode; - } - - public NodeContext getNodeContext() { - return nodeContext; - } - - public void setNodeContext(NodeContext nodeContext) { - this.nodeContext = nodeContext; - } - - public LongTermNodeExecution getNodeExecution() { - return nodeExecution; - } - - public void setNodeExecution(LongTermNodeExecution nodeExecution) { - this.nodeExecution = nodeExecution; - } - - public long getAskStatePeriod() { - return askStatePeriod; - } - - public void setAskStatePeriod(long askStatePeriod) { - this.askStatePeriod = askStatePeriod; - } - - public boolean isCompleted() { - return isCompleted; - } - - public void setCompleted(boolean completed) { - isCompleted = completed; - } - - public long getMaxLoopTime() { - return maxLoopTime; - } - - public void setMaxLoopTime(long maxLoopTime) { - this.maxLoopTime = maxLoopTime; - } - - public long getStartTime() { - return startTime; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/CompletedNodeExecutionResponse.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/CompletedNodeExecutionResponse.java deleted file mode 100644 index b2cbb57d9..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/CompletedNodeExecutionResponse.java +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public class CompletedNodeExecutionResponse implements NodeExecutionResponse { - - private boolean isSucceed; - - private String errorMsg; - - private String succeedMsg; - - private Exception exception; - - public void setIsSucceed(boolean isSucceed){ - this.isSucceed = isSucceed; - } - - public boolean isSucceed() { - return this.isSucceed; - } - - public String getErrorMsg(){ - return this.errorMsg; - } - - public void setErrorMsg(String errorMsg){ - this.errorMsg = errorMsg; - } - - - public String getSucceedMsg() { - return succeedMsg; - } - - public void setSucceedMsg(String succeedMsg) { - this.succeedMsg = succeedMsg; - } - - public Exception getException() { - return exception; - } - - public void setException(Exception exception) { - this.exception = exception; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/LongTermNodeExecutionAction.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/LongTermNodeExecutionAction.java deleted file mode 100644 index 2c798182b..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/LongTermNodeExecutionAction.java +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -/** - * Created by enjoyyin on 2019/11/18. - */ -public interface LongTermNodeExecutionAction { - void setSchedulerId(int schedulerId); - int getSchedulerId(); -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionAction.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionAction.java deleted file mode 100644 index a7414e516..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionAction.java +++ /dev/null @@ -1,25 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -/** - * created by enjoyyin on 2019/9/26 - * Description: - */ -public interface NodeExecutionAction { -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionResponse.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionResponse.java deleted file mode 100644 index ed6f3441c..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionResponse.java +++ /dev/null @@ -1,25 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public interface NodeExecutionResponse { -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionState.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionState.java deleted file mode 100644 index 19a4968cb..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/common/NodeExecutionState.java +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.common; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public enum NodeExecutionState { - /** - * NodeExecution的状态枚举 - */ - Accepted(0, "Accepted"), - Running(1, "Running"), - Success(2, "Success"), - Failed(3, "Failed"), - Killed(4, "Killed"), - Alert(5, "Alert"); - - private int code; - private String status; - - private NodeExecutionState(int code, String status){ - this.code = code; - this.status = status; - } - - public int getCode() { - return code; - } - - public String getStatus() { - return status; - } - - public static boolean isCompleted(NodeExecutionState state){ - return Success.equals(state) || Failed.equals(state) || Killed.equals(state); - } - - public static boolean isCompleted(String state){ - return Success.status.equals(state) || - Failed.status.equals(state) || - Killed.status.equals(state); - } - - public boolean isCompleted(){ - return isCompleted(this); - } - - public boolean isSuccess(){ - return Success.equals(this); - } - - - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/AbstractAppJointNode.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/AbstractAppJointNode.java deleted file mode 100644 index dad4d22b5..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/AbstractAppJointNode.java +++ /dev/null @@ -1,144 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import java.util.Map; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public abstract class AbstractAppJointNode implements AppJointNode{ - - String projectName; - - long projectId; - - - String flowName; - - long flowId; - - String nodeName; - - String nodeId; - - String nodeType; - - Map jobContent; - - - public AbstractAppJointNode(String projectName, long projectId, String flowName, long flowId, - String nodeName, String nodeTye, Map jobContent){ - this.projectId = projectId; - this.projectName = projectName; - this.flowName = flowName; - this.flowId = flowId; - this.nodeName = nodeName; - this.nodeType = nodeTye; - this.jobContent = jobContent; - } - - public AbstractAppJointNode(){ - - } - - - public void setProjectName(String projectName) { - this.projectName = projectName; - } - - public void setProjectId(long projectId) { - this.projectId = projectId; - } - - public void setFlowName(String flowName) { - this.flowName = flowName; - } - - public void setFlowId(long flowId) { - this.flowId = flowId; - } - - - @Override - public String getId() { - return this.nodeId; - } - - @Override - public void setId(String id) { - this.nodeId = id; - } - - @Override - public String getNodeType() { - return this.nodeType; - } - - @Override - public void setNodeType(String nodeType) { - this.nodeType = nodeType; - } - - @Override - public String getName() { - return this.nodeName; - } - - @Override - public void setName(String name) { - this.nodeName = name; - } - - @Override - public long getProjectId() { - return this.projectId; - } - - - - - - - @Override - public String getProjectName() { - return this.projectName; - } - - @Override - public String getFlowName() { - return this.flowName; - } - - @Override - public long getFlowId() { - return this.flowId; - } - - @Override - public Map getJobContent() { - return this.jobContent; - } - - @Override - public void setJobContent(Map jobContent) { - this.jobContent = jobContent; - } - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/AppJointNode.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/AppJointNode.java deleted file mode 100644 index 8295026c0..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/AppJointNode.java +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.dss.common.entity.node.Node; - -import java.util.Map; - -/** - * created by enjoyyin on 2019/9/25 - * Description: - */ -public interface AppJointNode extends Node { - /** - * 获取的node所属的project的id - * @return - */ - long getProjectId(); - - /** - * 获取node所属的project的name - * @return - */ - - String getProjectName(); - String getFlowName(); - long getFlowId(); - - /** - * 获取到Node的执行内容,执行内容是以Map的形式的 - * @return - */ - Map getJobContent(); - void setJobContent(Map jobContent); -} - diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/CallbackLongTermNodeExecution.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/CallbackLongTermNodeExecution.java deleted file mode 100644 index d5389504f..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/CallbackLongTermNodeExecution.java +++ /dev/null @@ -1,59 +0,0 @@ - -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - - -import com.webank.wedatasphere.dss.appjoint.execution.common.AsyncNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.common.CompletedNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; -import com.webank.wedatasphere.dss.appjoint.execution.conf.NodeExecutionConfiguration; - -import java.util.Map; -import java.util.concurrent.ConcurrentHashMap; - -/** - * created by enjoyyin on 2019/9/26 - * Description: - */ -public abstract class CallbackLongTermNodeExecution extends LongTermNodeExecution { - - protected Map asyncResponses = new ConcurrentHashMap(); - - public String getCallbackURL(NodeContext nodeContext){ - String urlSuffix = NodeExecutionConfiguration.CALL_BACK_URL().getValue(); - return nodeContext.getGatewayUrl() + urlSuffix; - } - - public abstract void acceptCallback(Map callbackMap); - - protected void markCompleted(NodeExecutionAction action, CompletedNodeExecutionResponse resultResponse) { - AsyncNodeExecutionResponse response = asyncResponses.get(action); - asyncResponses.remove(action); - response.getListeners().forEach(l -> l.onNodeExecutionCompleted(resultResponse)); - response.setCompleted(true); - } - - @Override - protected AsyncNodeExecutionResponse createAsyncNodeExecutionResponse(AppJointNode appJointNode, NodeContext context, NodeExecutionAction action) { - AsyncNodeExecutionResponse response = super.createAsyncNodeExecutionResponse(appJointNode, context, action); - response.setAskStatePeriod(NodeExecutionConfiguration.CALLBACK_NODE_EXECUTION_REFRESH_INTERVAL().getValue().toLong()); - asyncResponses.put(action, response); - return response; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/CommonAppJointNode.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/CommonAppJointNode.java deleted file mode 100644 index e3e352f82..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/CommonAppJointNode.java +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - -/** - * created by enjoyyin on 2019/10/9 - * Description: - */ -public class CommonAppJointNode extends AbstractAppJointNode{ - - private Map params; - - - private List dependencys = new ArrayList<>(); - - public CommonAppJointNode(String projectName, long projectId , String flowName, long flowId, String nodeName, - String type, Map jobContent){ - super(projectName, projectId, flowName, flowId, nodeName, type, jobContent); - } - - public CommonAppJointNode(){ - - } - - - public void setParams(Map params){ - this.params = params; - } - - - - - @Override - public void addDependency(String nodeName) { - this.dependencys.add(nodeName); - } - - @Override - public void setDependency(List dependencys) { - this.dependencys = dependencys; - } - - @Override - public void removeDependency(String nodeName) { - this.dependencys.remove(nodeName); - } - - @Override - public List getDependencys() { - return this.dependencys; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/HttpLongTermNodeExecution.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/HttpLongTermNodeExecution.java deleted file mode 100644 index e33dee7cc..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/HttpLongTermNodeExecution.java +++ /dev/null @@ -1,9 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.execution.core; - -/** - * created by enjoyyin on 2019/11/26 - * Description: - */ -public abstract class HttpLongTermNodeExecution extends LongTermNodeExecution{ - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/Killable.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/Killable.java deleted file mode 100644 index 10f0a3a6d..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/Killable.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; - -/** - * created by enjoyyin on 2019/9/26 - * Description: - */ -public interface Killable { - public boolean kill(NodeExecutionAction action); -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/LongTermNodeExecution.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/LongTermNodeExecution.java deleted file mode 100644 index b56d53c85..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/LongTermNodeExecution.java +++ /dev/null @@ -1,94 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution; -import com.webank.wedatasphere.dss.appjoint.execution.async.NodeExecutionListener; -import com.webank.wedatasphere.dss.appjoint.execution.common.*; -import com.webank.wedatasphere.dss.appjoint.execution.scheduler.LongTermNodeExecutionScheduler; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.ArrayList; -import java.util.List; - -/** - * created by enjoyyin on 2019/9/26 - * Description: LongTermNodeExecution 是一个 - */ -public abstract class LongTermNodeExecution implements NodeExecution { - - private static final Logger logger = LoggerFactory.getLogger(LongTermNodeExecution.class); - - private List nodeExecutionListener = new ArrayList(); - private LongTermNodeExecutionScheduler scheduler = SchedulerManager.getScheduler(); - - public void addNodeExecutionListener(NodeExecutionListener nodeExecutionListener) { - this.nodeExecutionListener.add(nodeExecutionListener); - } - - public LongTermNodeExecutionScheduler getScheduler() { - return scheduler; - } - - public void setScheduler(LongTermNodeExecutionScheduler scheduler) { - this.scheduler = scheduler; - } - - protected abstract NodeExecutionAction submit(AppJointNode appJointNode, NodeContext nodeContext, Session session); - - public abstract NodeExecutionState state(NodeExecutionAction action); - - public abstract CompletedNodeExecutionResponse result(NodeExecutionAction action, NodeContext nodeContext); - - - - @Override - public NodeExecutionResponse execute(AppJointNode appJointNode, NodeContext context, Session session) { - nodeExecutionListener.forEach(l -> l.beforeSubmit(appJointNode, context)); - NodeExecutionAction action = submit(appJointNode, context, session); - nodeExecutionListener.forEach(l -> l.afterSubmit(appJointNode, context, action)); - NodeExecutionState state = state(action); - if(state != null && state.isCompleted()) { - CompletedNodeExecutionResponse response = result(action, context); - nodeExecutionListener.forEach(l -> l.afterCompletedNodeExecutionResponse(appJointNode, context, action, response)); - return response; - } else { - AsyncNodeExecutionResponse response = createAsyncNodeExecutionResponse(appJointNode, context, action); - response.setNodeExecution(this); - response.addListener(r -> { - nodeExecutionListener.forEach(l -> l.afterCompletedNodeExecutionResponse(appJointNode, context, action, r)); - }); - nodeExecutionListener.forEach(l -> l.afterAsyncNodeExecutionResponse(response)); - if(scheduler != null) { - scheduler.addAsyncResponse(response); - } - return response; - } - } - - protected AsyncNodeExecutionResponse createAsyncNodeExecutionResponse(AppJointNode appJointNode, NodeContext context, NodeExecutionAction action) { - AsyncNodeExecutionResponse response = new AsyncNodeExecutionResponse(); - response.setAction(action); - response.setAppJointNode(appJointNode); - response.setNodeContext(context); - return response; - } - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/NodeContext.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/NodeContext.java deleted file mode 100644 index 1d86103a6..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/NodeContext.java +++ /dev/null @@ -1,112 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.linkis.common.io.FsPath; -import com.webank.wedatasphere.linkis.common.io.MetaData; -import com.webank.wedatasphere.linkis.common.io.Record; -import com.webank.wedatasphere.linkis.common.io.resultset.ResultSet; -import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetReader; -import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetWriter; -import com.webank.wedatasphere.linkis.protocol.query.RequestPersistTask; -import com.webank.wedatasphere.linkis.storage.LineMetaData; -import com.webank.wedatasphere.linkis.storage.LineRecord; -import com.webank.wedatasphere.linkis.storage.resultset.table.TableMetaData; -import com.webank.wedatasphere.linkis.storage.resultset.table.TableRecord; - -import java.util.HashMap; -import java.util.Map; - -/** - * created by enjoyyin on 2019/10/12 - * Description: - */ -public interface NodeContext { - - - static final String TOKEN_USER_STR = "Token-User"; - - static final String TOKEN_CODE_STR = "Token-Code"; - - - public boolean isReadNode(); - - public AppJointNode getAppJointNode(); - - public Map getRuntimeMap(); - - public long[] getJobIdsOfShareNode(); - - public RequestPersistTask getJobById(long jobId); - - - //获取任务的所有的结果集路径 - FsPath[] getResultSetPathsByJobId(long jobId); - - public void appendLog(String log); - - public void updateProgress(float progress); - - /** - * 获取本节点的操作用户 - * @return - */ - public String getUser(); - - void setStorePath(String storePath); - - String getStorePath(); - - ResultSetWriter createTableResultSetWriter(); - - ResultSetWriter createTableResultSetWriter(String resultSetAlias); - - ResultSetWriter createTextResultSetWriter(); - - ResultSetWriter createTextResultSetWriter(String resultSetAlias); - - ResultSetWriter createHTMLResultSetWriter(); - - ResultSetWriter createHTMLResultSetWriter(String resultSetAlias); - - ResultSetWriter createPictureResultSetWriter(); - - ResultSetWriter createPictureResultSetWriter(String resultSetAlias); - - - - ResultSetWritercreateResultSetWriter(ResultSet resultSet, String resultSetAlias); - - - public ResultSetReader getResultSetReader(FsPath fsPath); - - /** - * - * @return - */ - public String getGatewayUrl(); - - - default Map getTokenHeader(String user){ - Map tokenHeader = new HashMap<>(); - tokenHeader.put(TOKEN_CODE_STR, "dss-AUTH"); - tokenHeader.put(TOKEN_USER_STR, user); - return tokenHeader; - } - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/NodeContextImpl.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/NodeContextImpl.java deleted file mode 100644 index 99ab15701..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/NodeContextImpl.java +++ /dev/null @@ -1,94 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - -import java.util.Map; - -/** - * created by enjoyyin on 2019/10/12 - * Description: - */ -public abstract class NodeContextImpl implements NodeContext{ - - - private boolean isReadNode; - - private AppJointNode appJointNode; - - private Map runtimeMap; - - private long[] jobIdsOfShareNode; - - private String user; - - private Session session; - - - public void setIsReadNode(boolean isReadNode){ - this.isReadNode = isReadNode; - } - - public void setAppJointNode(AppJointNode appJointNode){ - this.appJointNode = appJointNode; - } - - public void setRuntimeMap(Map runtimeMap){ - this.runtimeMap = runtimeMap; - } - - public void setJobIdsOfShareNode(long[] jobIds){ - this.jobIdsOfShareNode = jobIds; - } - - public void setUser(String user){ - this.user = user; - } - - - - @Override - public boolean isReadNode() { - return this.isReadNode; - } - - @Override - public AppJointNode getAppJointNode() { - return this.appJointNode; - } - - @Override - public Map getRuntimeMap() { - return this.runtimeMap; - } - - @Override - public long[] getJobIdsOfShareNode() { - return this.jobIdsOfShareNode; - } - - @Override - public String getUser() { - return this.user; - } - - - - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/Procedure.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/Procedure.java deleted file mode 100644 index 726177cc3..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/Procedure.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; - -/** - * created by enjoyyin on 2019/9/29 - * Description: - */ -public interface Procedure { - - public float progress(NodeExecutionAction action); - - public String log(NodeExecutionAction action); - - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/SchedulerManager.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/SchedulerManager.java deleted file mode 100644 index a51dee9d8..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/SchedulerManager.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.core; - -import com.webank.wedatasphere.dss.appjoint.execution.scheduler.ListenerEventBusNodeExecutionScheduler; -import com.webank.wedatasphere.dss.appjoint.execution.scheduler.LongTermNodeExecutionScheduler; - -/** - * created by enjoyyin on 2019/11/15 - * Description: - */ -public class SchedulerManager { - - private static LongTermNodeExecutionScheduler scheduler = new ListenerEventBusNodeExecutionScheduler(); - - static{ - ((ListenerEventBusNodeExecutionScheduler)scheduler).start(); - } - - public static LongTermNodeExecutionScheduler getScheduler(){ - return scheduler; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/AsyncNodeExecutionResponseEvent.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/AsyncNodeExecutionResponseEvent.java deleted file mode 100644 index c35108d06..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/AsyncNodeExecutionResponseEvent.java +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.scheduler; - -import com.webank.wedatasphere.dss.appjoint.execution.common.AsyncNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.core.LongTermNodeExecution; -import com.webank.wedatasphere.linkis.common.listener.Event; - -/** - * Created by enjoyyin on 2019/11/13. - */ -public class AsyncNodeExecutionResponseEvent implements Event { - - private AsyncNodeExecutionResponse response; - private long lastAskTime = 0; - - public AsyncNodeExecutionResponseEvent(AsyncNodeExecutionResponse response) { - this.response = response; - } - - public AsyncNodeExecutionResponse getResponse() { - return response; - } - - public void setResponse(AsyncNodeExecutionResponse response) { - this.response = response; - } - - public long getLastAskTime() { - return lastAskTime; - } - - public void setLastAskTime() { - this.lastAskTime = System.currentTimeMillis(); - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/AsyncNodeExecutionSchedulerListener.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/AsyncNodeExecutionSchedulerListener.java deleted file mode 100644 index ff33af2b8..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/AsyncNodeExecutionSchedulerListener.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.scheduler; - -import com.webank.wedatasphere.linkis.common.listener.Event; -import com.webank.wedatasphere.linkis.common.listener.EventListener; - -/** - * Created by enjoyyin on 2019/11/13. - */ -public interface AsyncNodeExecutionSchedulerListener extends EventListener { - - void onEvent(AsyncNodeExecutionResponseEvent event); - - void onEventError(AsyncNodeExecutionResponseEvent event, Throwable t); - - @Override - default void onEventError(Event event, Throwable t) { - if(event instanceof AsyncNodeExecutionResponseEvent) { - onEventError((AsyncNodeExecutionResponseEvent) event, t); - } - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/LongTermNodeExecutionScheduler.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/LongTermNodeExecutionScheduler.java deleted file mode 100644 index 99430ddea..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/scheduler/LongTermNodeExecutionScheduler.java +++ /dev/null @@ -1,39 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.scheduler; - -import com.webank.wedatasphere.dss.appjoint.execution.common.AsyncNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.common.LongTermNodeExecutionAction; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; - -/** - * Created by enjoyyin on 2019/11/13. - */ -public interface LongTermNodeExecutionScheduler { - - void addAsyncResponse(AsyncNodeExecutionResponse response); - - void removeAsyncResponse(LongTermNodeExecutionAction action); - - AsyncNodeExecutionResponse getAsyncResponse(LongTermNodeExecutionAction action); - - void start(); - - void stop(); - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/AppJointUrl.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/AppJointUrl.java deleted file mode 100644 index d3b00c7e7..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/AppJointUrl.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.service; - -/** - * Created by enjoyyin on 2019/11/5. - */ -public interface AppJointUrl { - - String getBaseUrl(); - - void setBaseUrl(String basicUrl); - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/AppJointUrlImpl.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/AppJointUrlImpl.java deleted file mode 100644 index 017de5b5e..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/AppJointUrlImpl.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.service; - -/** - * Created by enjoyyin on 2019/11/5. - */ -public abstract class AppJointUrlImpl implements AppJointUrl { - - private String baseUrl; - - @Override - public String getBaseUrl() { - return baseUrl; - } - - @Override - public void setBaseUrl(String basicUrl) { - this.baseUrl = basicUrl; - } -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/NodeService.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/NodeService.java deleted file mode 100644 index 3a859b00e..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/NodeService.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.service; - -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - -/** - * Created by enjoyyin on 2019/10/10. - */ -public interface NodeService extends AppJointUrl { - - /** - * 根据参数创建外部节点 - * @param session 连接会话 - * @param node 节点信息 - * @param requestBody 创建节点需要的参数，前端带过来的Json结构字符串。 - * @return 返回jobContent的Map，工作流会将该Map存储起来，作为该节点的关键关联信息，用于后续的CRUD和执行。 - */ - Map createNode(Session session, AppJointNode node, - Map requestBody) throws AppJointErrorException; - - void deleteNode(Session session, AppJointNode node) throws AppJointErrorException; - - Map updateNode(Session session, AppJointNode node, Map requestBody) throws AppJointErrorException; - - default Map refresh(Session session, AppJointNode node) throws AppJointErrorException { - return null; - } - - default void copyNode(Session session, AppJointNode newNode, AppJointNode oldNode) throws AppJointErrorException { } - - default void setNodeReadOnly(Session session, AppJointNode node) throws AppJointErrorException {} - - default List listNodes(Session session, AppJointNode node) throws AppJointErrorException { - return new ArrayList<>(); - } - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/ProjectService.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/ProjectService.java deleted file mode 100644 index 30edffd84..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/ProjectService.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.service; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.project.Project; - -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface ProjectService extends AppJointUrl { - - List fetchProjects(Session session) throws AppJointErrorException; - - Project createProject(Project project, Session session) throws AppJointErrorException; - - void deleteProject(Project project, Session session) throws AppJointErrorException; - - default Project getProject(Project project, Session session) throws AppJointErrorException { - return null; - } - - void updateProject(Project project, Session session) throws AppJointErrorException; -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/SecurityService.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/SecurityService.java deleted file mode 100644 index a1a035032..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/SecurityService.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.service; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface SecurityService extends AppJointUrl { - - Session login(String user) throws AppJointErrorException; - - void logout(String user) throws AppJointErrorException; - -} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/session/Session.java b/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/session/Session.java deleted file mode 100644 index 4f0f95674..000000000 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/service/session/Session.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.service.session; - -import org.apache.http.Header; -import org.apache.http.cookie.Cookie; - -import java.util.Map; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface Session { - - String getUser(); - - Cookie[] getCookies(); - - Header[] getHeaders(); - - Map getParameters(); - - long getLastAccessTime(); - - void updateLastAccessTime(); -} diff --git a/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/conf/NodeExecutionConfiguration.scala b/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/conf/NodeExecutionConfiguration.scala deleted file mode 100644 index dca56c995..000000000 --- a/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/conf/NodeExecutionConfiguration.scala +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.conf - -import com.webank.wedatasphere.linkis.common.conf.{CommonVars, TimeType} - -/** - * created by enjoyyin on 2019/9/29 - * Description: - */ -object NodeExecutionConfiguration { - val QUERY_APPLICATION_NAME = CommonVars("wds.dss.appjoint.query.service.name", "cloud-query") - val CALL_BACK_URL = CommonVars("wds.dss.appjoint.callback.url", "/api/rest_j/v1/entrance/callback") - - val ASYNC_NODE_EXECUTION_SCHEDULER_QUEUE_SIZE = CommonVars("wds.dss.appjoint.nodeexecution.scheduler.queue.size", 2000) - val ASYNC_NODE_EXECUTION_SCHEDULER_THREAD_SIZE = CommonVars("wds.dss.appjoint.nodeexecution.scheduler.thread.max", 20) - - val CALLBACK_NODE_EXECUTION_REFRESH_INTERVAL = CommonVars("wds.dss.appjoint.nodeexecution.callback.refresh.interval", new TimeType("2m")) - - val WORKFLOW_SHARED_NODES_JOBIDS = "workflow.shared.nodes.jobids" -} diff --git a/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/context/NodeExecutionContext.scala b/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/context/NodeExecutionContext.scala deleted file mode 100644 index 7381bae2c..000000000 --- a/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/context/NodeExecutionContext.scala +++ /dev/null @@ -1,81 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.context - -import com.webank.wedatasphere.linkis.common.utils.Logging - -/** - * created by enjoyyin on 2019/9/27 - * Description: - */ -object NodeExecutionContext extends Logging{ - -// private lazy val querySender = Sender.getSender(NodeExecutionConfiguration.QUERY_APPLICATION_NAME.getValue) -// -// val SUCCESS_FLAG:Int = 0 -// -// val TASK_MAP_KEY:String = "task" -// -// /** -// * 通过的taskId将所有的node都拿到 -// * @param taskIds -// * @return -// */ -// -// def getNodes(taskIds:Array[Long]):Array[AppJointNode] = { -// val nodes = new ArrayBuffer[AppJointNode]() -// val requestPersistTasks = new ArrayBuffer[RequestPersistTask]() -// taskIds foreach { -// taskId => val requestQueryTask = new RequestQueryTask() -// requestQueryTask.setTaskID(taskId) -// Utils.tryCatch{ -// val taskResponse = querySender.ask(requestQueryTask) -// taskResponse match { -// case responsePersist:ResponsePersist => val status = responsePersist.getStatus -// if (status != SUCCESS_FLAG){ -// logger.error(s"query from jobHistory status failed, status is $status") -// throw new AppJointErrorException(75533, "query from jobHistory status failed") -// }else{ -// val data = responsePersist.getData -// data.get(TASK_MAP_KEY) match { -// case tasks:util.List[RequestPersistTask] => tasks.get(0) match { -// case requestPersistTask:RequestPersistTask => requestPersistTasks += requestPersistTask -// case _ => logger.error(s"query from jobhistory not a correct RequestPersistTask type taskId is $taskId") -// throw new AppJointErrorException(75533, s"query from jobhistory not a correct RequestPersistTask type taskId is $taskId") -// } -// case _ => logger.error(s"query from jobhistory not a correct List type taskId is $taskId") -// throw new AppJointErrorException(75533, s"query from jobhistory not a correct List type taskId is $taskId") -// } -// } -// case _ => logger.error("get query response incorrectly") -// throw new AppJointErrorException(75533, "get query response incorrectly") -// } -// }{ -// case errorException:ErrorException => logger.error(s"query taskId $taskId error", errorException) -// throw errorException -// case e:Exception => logger.error(s"query taskId $taskId error", e) -// throw new AppJointErrorException(75533, s"query taskId $taskId error") -// } -// } -// requestPersistTasks foreach { -// requestPersistTask => requestPersistTask.getSource -// } -// nodes.toArray -// } - -} diff --git a/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/scheduler/ListenerEventBusNodeExecutionScheduler.scala b/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/scheduler/ListenerEventBusNodeExecutionScheduler.scala deleted file mode 100644 index 51d167eaf..000000000 --- a/dss-appjoint-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/execution/scheduler/ListenerEventBusNodeExecutionScheduler.scala +++ /dev/null @@ -1,125 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.execution.scheduler - -import java.util.concurrent.ArrayBlockingQueue - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException -import com.webank.wedatasphere.dss.appjoint.execution.common.{AsyncNodeExecutionResponse, CompletedNodeExecutionResponse, LongTermNodeExecutionAction} -import com.webank.wedatasphere.dss.appjoint.execution.conf.NodeExecutionConfiguration._ -import com.webank.wedatasphere.linkis.common.collection.BlockingLoopArray -import com.webank.wedatasphere.linkis.common.listener.ListenerEventBus -import com.webank.wedatasphere.linkis.common.utils.{ByteTimeUtils, Utils} -import org.apache.commons.lang.time.DateFormatUtils - -import scala.collection.JavaConversions._ - -/** - * Created by enjoyyin on 2019/11/13. - */ -class ListenerEventBusNodeExecutionScheduler(eventQueueCapacity: Int, name: String)(listenerConsumerThreadSize: Int) - extends LongTermNodeExecutionScheduler { - - private val listenerEventBus = new ListenerEventBus[AsyncNodeExecutionSchedulerListener, AsyncNodeExecutionResponseEvent](eventQueueCapacity, name)(listenerConsumerThreadSize) { - override protected val dropEvent: DropEvent = new DropEvent { - override def onDropEvent(event: AsyncNodeExecutionResponseEvent): Unit = throw new AppJointErrorException(95536, "LongTermNodeExecutionScheduler is full, please ask admin for help!") - override def onBusStopped(event: AsyncNodeExecutionResponseEvent): Unit = throw new AppJointErrorException(95536, "LongTermNodeExecutionScheduler is stopped, please ask admin for help!") - } - override def doPostEvent(listener: AsyncNodeExecutionSchedulerListener, event: AsyncNodeExecutionResponseEvent): Unit = { - listener.onEvent(event) - } - } - - def this() = { - this(ASYNC_NODE_EXECUTION_SCHEDULER_QUEUE_SIZE.getValue, "Async-NodeExecution-Scheduler")(ASYNC_NODE_EXECUTION_SCHEDULER_THREAD_SIZE.getValue) - getAsyncNodeExecutionSchedulerListeners.foreach(listenerEventBus.addListener) - } - - private val eventQueue = { - val ru = scala.reflect.runtime.universe - val classMirror = ru.runtimeMirror(getClass.getClassLoader) - val listenerEventBusClass = classMirror.reflect(listenerEventBus) - val field1 = ru.typeOf[ListenerEventBus[_, _]].decl(ru.TermName("eventQueue")).asMethod - val result = listenerEventBusClass.reflectMethod(field1) - result() match { - case queue: ArrayBlockingQueue[AsyncNodeExecutionResponseEvent] => queue - } - } - - protected def getAsyncNodeExecutionSchedulerListeners: Array[AsyncNodeExecutionSchedulerListener] = { - Array(new AsyncNodeExecutionSchedulerListener() { - override def onEvent(event: AsyncNodeExecutionResponseEvent): Unit = if(!event.getResponse.isCompleted) { - val response = event.getResponse - if(response.getMaxLoopTime > 0 && System.currentTimeMillis - response.getStartTime >= response.getMaxLoopTime) { - onEventError(event, new AppJointErrorException(75533, s"AppJointNode Execution is overtime! StartTime is ${DateFormatUtils.format(response.getStartTime, "yyyy-MM-dd HH:mm:ss")}, maxWaitTime is " + - ByteTimeUtils.msDurationToString(response.getMaxLoopTime))) - return - } - val period = System.currentTimeMillis() - event.getLastAskTime - if(period < response.getAskStatePeriod) { - if(period < 10) Utils.sleepQuietly(100) - if(!response.isCompleted) addEvent(event) - return - } - val state = response.getNodeExecution.state(response.getAction) - if(state.isCompleted) { - val resultResponse = response.getNodeExecution.result(response.getAction, response.getNodeContext) - onEventCompleted(event, resultResponse) - } else if(!response.isCompleted) { - event.setLastAskTime() - addEvent(event) - } - } - - private def onEventCompleted(event: AsyncNodeExecutionResponseEvent, response: CompletedNodeExecutionResponse): Unit = { - event.getResponse.getListeners.foreach(_.onNodeExecutionCompleted(response)) - event.getResponse.setCompleted(true) - } - - override def onEventError(event: AsyncNodeExecutionResponseEvent, t: Throwable): Unit = t match { - case e: Exception => - val response = new CompletedNodeExecutionResponse - response.setIsSucceed(false) - response.setException(e) - onEventCompleted(event, response) - } - }) - } - - override def addAsyncResponse(response: AsyncNodeExecutionResponse): Unit = - addEvent(new AsyncNodeExecutionResponseEvent(response)) - - protected def addEvent(event: AsyncNodeExecutionResponseEvent): Unit = synchronized { - listenerEventBus.post(event) -// event.getResponse.getAction match { -// case longTermAction: LongTermNodeExecutionAction => -// longTermAction.setSchedulerId(eventQueue.max) -// case _ => -// } - } - - override def removeAsyncResponse(action: LongTermNodeExecutionAction): Unit = { - - } - - override def getAsyncResponse(action: LongTermNodeExecutionAction): AsyncNodeExecutionResponse = null - - override def start(): Unit = listenerEventBus.start() - - override def stop(): Unit = listenerEventBus.stop() -} \ No newline at end of file diff --git a/dss-appjoint-loader/pom.xml b/dss-appjoint-loader/pom.xml deleted file mode 100644 index 99ab6ac8b..000000000 --- a/dss-appjoint-loader/pom.xml +++ /dev/null @@ -1,71 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-appjoint-loader - 0.9.1 - - - - com.webank.wedatasphere.dss - dss-appjoint-core - ${dss.version} - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - **/*.yml - **/*.properties - **/*.sh - **/log4j2.xml - - - - - - - src/main/resources - - - - - - \ No newline at end of file diff --git a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/clazzloader/AppJointClassLoader.java b/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/clazzloader/AppJointClassLoader.java deleted file mode 100644 index c0c04dde6..000000000 --- a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/clazzloader/AppJointClassLoader.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.clazzloader; - -import com.google.common.collect.ImmutableList; -import com.google.common.collect.Iterables; -import com.webank.wedatasphere.dss.appjoint.loader.AppJointLoader; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.lang.reflect.InvocationTargetException; -import java.lang.reflect.Method; -import java.net.URL; -import java.net.URLClassLoader; -import java.util.List; -import java.util.ServiceLoader; - -import static java.util.Objects.requireNonNull; - -/** - * created by cooperyang on 2019/11/8 - * Description: - */ -public class AppJointClassLoader extends URLClassLoader{ - - private static final Logger logger = LoggerFactory.getLogger(AppJointClassLoader.class); - - - - public AppJointClassLoader(URL[] urls, ClassLoader parent) { - super(urls, parent); - } - - - @Override - public Class loadClass(String name) throws ClassNotFoundException { - return super.loadClass(name); - } - - @Override - protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { - return super.loadClass(name, resolve); - } -} diff --git a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/AppJointLoader.java b/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/AppJointLoader.java deleted file mode 100644 index 5b890333a..000000000 --- a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/AppJointLoader.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.loader; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.linkis.common.exception.ErrorException; - -import java.util.Map; -/** - * created by cooperyang on 2019/11/8 - * Description: AppJointLoader 是为了上层模块，如server，entrance等模块能够获取到指定的AppJoint - */ -public interface AppJointLoader { - - - String DIR_NAME = "appjoints"; - - String APPJOINT_DIR_NAME = "dss-appjoints"; - - String PROPERTIES_NAME = "appjoint.properties"; - - String CONF_NAME = "conf"; - - String LIB_NAME = "lib"; - - String JAR_SUF_NAME = ".jar"; - - String FILE_SCHEMA = "file://"; - - /** - * 获取一个appjoint,需要通过baseUrl,name 和 params进行获取 - * @param baseUrl appjoint代理的外部系统的url - * @return 一个appjoint - */ - - AppJoint getAppJoint(String baseUrl, String appJointName, Map params) throws Exception; - - - static AppJointLoader getAppJointLoader(){ - return AppJointLoaderFactory.getAppJointLoader(); - } - - -} diff --git a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/AppJointLoaderFactory.java b/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/AppJointLoaderFactory.java deleted file mode 100644 index fead09f7c..000000000 --- a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/AppJointLoaderFactory.java +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.loader; - -import com.webank.wedatasphere.dss.appjoint.conf.AppJointLoaderConf; -import org.apache.commons.lang.ClassUtils; -import org.apache.commons.lang.StringUtils; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * created by cooperyang on 2019/11/8 - * Description: - */ -class AppJointLoaderFactory { - - - private static Class clazz = CommonAppJointLoader.class; - private static AppJointLoader appJointLoader = null; - private static final Logger logger = LoggerFactory.getLogger(AppJointLoaderFactory.class); - - - static AppJointLoader getAppJointLoader(){ - if (appJointLoader == null){ - synchronized (AppJointLoaderFactory.class){ - if (appJointLoader == null){ - String className = AppJointLoaderConf.CLASS_LOADER_CLASS_NAME().getValue(); - if (clazz == CommonAppJointLoader.class && StringUtils.isNotBlank(className)){ - try{ - clazz = ClassUtils.getClass(className); - }catch(ClassNotFoundException e){ - logger.warn("can not get class {}", className, e); - } - try { - appJointLoader = clazz.newInstance(); - } catch (Exception e) { - logger.error("can not initialize class", e); - } - }else{ - try { - appJointLoader = clazz.newInstance(); - } catch (Exception e) { - logger.error("can not initialize class", e); - } - } - } - } - } - return appJointLoader; - } -} diff --git a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/CommonAppJointLoader.java b/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/CommonAppJointLoader.java deleted file mode 100644 index 681f8de21..000000000 --- a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/loader/CommonAppJointLoader.java +++ /dev/null @@ -1,150 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.loader; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.clazzloader.AppJointClassLoader; -import com.webank.wedatasphere.dss.appjoint.utils.AppJointUtils; -import com.webank.wedatasphere.dss.appjoint.utils.ExceptionHelper; -import com.webank.wedatasphere.linkis.common.exception.ErrorException; -import org.apache.commons.lang.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.*; -import java.net.MalformedURLException; -import java.net.URL; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Properties; - -/** - * created by cooperyang on 2019/11/8 - * Description: - */ -public class CommonAppJointLoader implements AppJointLoader{ - - - private static final Logger logger = LoggerFactory.getLogger(CommonAppJointLoader.class); - - private final Map appJoints = new HashMap<>(); - - - /** - * 用来存放每一个appjoint所用到的classloader，这样的话，每一个appjoint的classloader都是不一样的 - */ - private final Map classLoaders = new HashMap<>(); - - - - /** - * - * @param baseUrl appjoint代理的外部系统的url - * @param appJointName appJoint的名字 - * @param params 参数用来进行init - * @return - * 命名规范必须是 ${DSS_HOME}/appjoints/${appjointName}/ - */ - @Override - public AppJoint getAppJoint(String baseUrl, String appJointName, Map params) throws Exception{ - synchronized (appJoints){ - if (appJoints.containsKey(appJointName)){ - return appJoints.get(appJointName); - } - } - if (params == null) { - params = new HashMap(); - } - ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader(); - ClassLoader newClassLoader = null; - synchronized (classLoaders){ - if (classLoaders.containsKey(appJointName)){ - newClassLoader = classLoaders.get(appJointName); - } - } - URL classpathUrl = oldClassLoader.getResource(""); - logger.info("classpathUrl is {}", classpathUrl.getPath()); - if(null == classpathUrl){ - throw new ErrorException(70059, "classPathUrl is null"); - } - - - - String basePathUrlStr = classpathUrl.getPath() + ".." + File.separator + ".." + File.separator + AppJointLoader.APPJOINT_DIR_NAME; - - String libPathUrlStr = basePathUrlStr + File.separator + appJointName + - File.separator + AppJointLoader.LIB_NAME; - String propertiesUrlStr = basePathUrlStr + File.separator + appJointName + - File.separator + AppJointLoader.PROPERTIES_NAME; - try{ - params.putAll(readFromProperties(propertiesUrlStr)); - }catch(IOException e){ - logger.warn("cannot get properties from {}", propertiesUrlStr, e); - } - URL finalURL = null; - try { - finalURL = new URL(AppJointLoader.FILE_SCHEMA + libPathUrlStr + "/*"); - } catch (MalformedURLException e) { - ExceptionHelper.dealErrorException(70061, libPathUrlStr + " url is wrong", e); - } - List jars = AppJointUtils.getJarsUrlsOfPath(libPathUrlStr); - if (newClassLoader == null){ - newClassLoader = new AppJointClassLoader(jars.toArray(new URL[100]) ,oldClassLoader); - } - synchronized (classLoaders){ - classLoaders.put(appJointName, newClassLoader); - } - Thread.currentThread().setContextClassLoader(newClassLoader); - String fullClassName = AppJointUtils.getAppJointClassName(appJointName, libPathUrlStr, newClassLoader); - Class clazz = null; - try { - clazz = newClassLoader.loadClass(fullClassName); - } catch (ClassNotFoundException e) { - ExceptionHelper.dealErrorException(70062, fullClassName + " class not found ", e); - } - if (clazz == null){ - Thread.currentThread().setContextClassLoader(oldClassLoader); - return null; - }else{ - AppJoint retAppjoint = (AppJoint) clazz.newInstance(); - if (StringUtils.isEmpty(baseUrl) && params.get("baseUrl") != null){ - baseUrl = params.get("baseUrl").toString(); - } - retAppjoint.init(baseUrl, params); - Thread.currentThread().setContextClassLoader(oldClassLoader); - synchronized (appJoints){ - appJoints.put(appJointName, retAppjoint); - } - logger.info("appJointName is {}, retAppJoint is {}", appJointName, retAppjoint.getClass().toString()); - return retAppjoint; - } - } - - private Map readFromProperties(String propertiesFile) throws IOException { - Properties properties = new Properties(); - BufferedReader reader = new BufferedReader(new FileReader(propertiesFile)); - properties.load(reader); - Map map = new HashMap((Map)properties); - return map; - } - - - - -} diff --git a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/utils/AppJointUtils.java b/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/utils/AppJointUtils.java deleted file mode 100644 index be7ceb0b3..000000000 --- a/dss-appjoint-loader/src/main/java/com/webank/wedatasphere/dss/appjoint/utils/AppJointUtils.java +++ /dev/null @@ -1,160 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.utils; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.exception.NoSuchAppJointException; -import com.webank.wedatasphere.dss.appjoint.loader.AppJointLoader; -import com.webank.wedatasphere.linkis.common.exception.ErrorException; -import org.apache.commons.lang.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.File; -import java.io.IOException; -import java.lang.reflect.Modifier; -import java.net.MalformedURLException; -import java.net.URL; -import java.util.ArrayList; -import java.util.Enumeration; -import java.util.List; -import java.util.jar.JarEntry; -import java.util.jar.JarFile; - -/** - * created by cooperyang on 2019/11/9 - * Description: - */ -public class AppJointUtils { - - - private static final Logger logger = LoggerFactory.getLogger(AppJointUtils.class); - - - private static Class PARENT_CLASS = AppJoint.class; - /** - * 通过appJointName去获取到相应的appjoint的全限定类名 - * @param appJointName appJointname - * @param libPath ${DSS_HOME}目录 - * @return appjointName对应的的appjoint的全限定类名 - */ - - public static String getAppJointClassName(String appJointName, String libPath, ClassLoader classLoader)throws NoSuchAppJointException { - //1.获取目录下面所有的jar包 - List jars = getJarsOfPath(libPath); - //2.从所有的jar中获取到AppJoint的子类 - boolean flag = false; - String appjoint = null; - for (String jar : jars){ - for(String clazzName : getClassNameFrom(jar)){ - if (isChildClass(clazzName, PARENT_CLASS, classLoader)){ - flag = true; - appjoint = clazzName; - break; - } - } - if (flag){ - break; - } - } - if (StringUtils.isEmpty(appjoint)){ - throw new NoSuchAppJointException(appJointName + " does not exist"); - } - return appjoint; - } - - - public static List getJarsOfPath(String path){ - File file = new File(path); - List jars = new ArrayList<>(); - if (file.listFiles() != null){ - for(File f : file.listFiles()){ - if (!f.isDirectory() && f.getName().endsWith(AppJointLoader.JAR_SUF_NAME) && f.getName().startsWith("dss")){ - jars.add(f.getPath()); - } - } - } - return jars; - } - - - public static List getJarsUrlsOfPath(String path){ - File file = new File(path); - List jars = new ArrayList<>(); - if (file.listFiles() != null){ - for(File f : file.listFiles()){ - if (!f.isDirectory() && f.getName().endsWith(AppJointLoader.JAR_SUF_NAME)){ - try { - jars.add(f.toURI().toURL()); - } catch (MalformedURLException e) { - logger.warn("url {} cannot be added", AppJointLoader.FILE_SCHEMA + f.getPath()); - } - } - } - } - return jars; - } - - - /** - * 从jar包读取所有的class文件名 - */ - private static List getClassNameFrom(String jarName) { - List fileList = new ArrayList(); - try { - JarFile jarFile = new JarFile(new File(jarName)); - Enumeration en = jarFile.entries(); - while (en.hasMoreElements()) { - String name1 = en.nextElement().getName(); - if (!name1.endsWith(".class")) { - continue; - } - String name2 = name1.substring(0, name1.lastIndexOf(".class")); - String name3 = name2.replaceAll("/", "."); - fileList.add(name3); - } - } catch (IOException e) { - e.printStackTrace(); - } - - return fileList; - } - - - private static boolean isChildClass(String className, Class parentClazz, ClassLoader classLoader) { - if (StringUtils.isEmpty(className)) { - return false; - } - Class clazz = null; - try { - clazz = classLoader.loadClass(className); - //忽略抽象类和接口 - if (Modifier.isAbstract(clazz.getModifiers())) { - return false; - } - if (Modifier.isInterface(clazz.getModifiers())) { - return false; - } - } catch (Throwable t) { - logger.error("className {} can not be instanced", className); - return false; - } - return parentClazz.isAssignableFrom(clazz); - } - -} diff --git a/dss-appjoint-loader/src/main/scala/com/webank/wedatasphere/dss/appjoint/conf/AppJointLoaderConf.scala b/dss-appjoint-loader/src/main/scala/com/webank/wedatasphere/dss/appjoint/conf/AppJointLoaderConf.scala deleted file mode 100644 index d20e96390..000000000 --- a/dss-appjoint-loader/src/main/scala/com/webank/wedatasphere/dss/appjoint/conf/AppJointLoaderConf.scala +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.conf - -import com.webank.wedatasphere.linkis.common.conf.CommonVars - -/** - * created by cooperyang on 2019/11/9 - * Description: - */ -object AppJointLoaderConf { - val CLASS_LOADER_CLASS_NAME = CommonVars("dss.appjoint.loader.classname", "") -} diff --git a/dss-appjoint-loader/src/main/scala/com/webank/wedatasphere/dss/appjoint/exception/NoSuchAppJointException.scala b/dss-appjoint-loader/src/main/scala/com/webank/wedatasphere/dss/appjoint/exception/NoSuchAppJointException.scala deleted file mode 100644 index 735b893ee..000000000 --- a/dss-appjoint-loader/src/main/scala/com/webank/wedatasphere/dss/appjoint/exception/NoSuchAppJointException.scala +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.exception - -import com.webank.wedatasphere.linkis.common.exception.ErrorException - -/** - * created by cooperyang on 2019/11/10 - * Description: - */ -case class NoSuchAppJointException(errDesc:String) extends ErrorException(70059, errDesc) diff --git a/dss-application/pom.xml b/dss-application/pom.xml deleted file mode 100644 index 8be0e0434..000000000 --- a/dss-application/pom.xml +++ /dev/null @@ -1,86 +0,0 @@ - - - - - 4.0.0 - - dss - com.webank.wedatasphere.dss - 0.9.1 - - dss-application - - - - junit - junit - 4.12 - test - - - - com.webank.wedatasphere.linkis - linkis-module - ${linkis.version} - provided - - - - com.webank.wedatasphere.dss - dss-appjoint-loader - ${dss.version} - - - org.apache.htrace - htrace-core - 3.2.0-incubating - compile - - - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - - - src/main/java - - **/*.xml - - - - - \ No newline at end of file diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/conf/ApplicationConf.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/conf/ApplicationConf.java deleted file mode 100644 index 1194eb427..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/conf/ApplicationConf.java +++ /dev/null @@ -1,52 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.conf; - - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; - -/** - * Created by chaogefeng on 2019/10/10. - */ -public class ApplicationConf { - - public static final CommonVars FAQ = CommonVars.apply("wds.linkis.application.dws.params",""); - - public static final String SUPER_USER_NAME = CommonVars.apply("wds.linkis.super.user.name","").getValue(); - public static final String WORKSPACE_USER_ROOT_PATH = CommonVars.apply("wds.linkis.workspace.user.root.path","").getValue(); - public static final String HDFS_USER_ROOT_PATH = CommonVars.apply("wds.linkis.hdfs.user.root.path","").getValue(); - public static final String RESULT_SET_ROOT_PATH = CommonVars.apply("wds.linkis.result.set.root.path","").getValue(); - public static final String WDS_SCHEDULER_PATH = CommonVars.apply("wds.linkis.scheduler.path","").getValue(); - public static final String WDS_USER_PATH = CommonVars.apply("wds.linkis.user.path","hdfs:///user").getValue(); - public static final String DSS_INSTALL_DIR = CommonVars.apply("wds.linkis.dss.install.dir","").getValue(); - public static final String AZKABAN_INSTALL_DIR = CommonVars.apply("wds.linkis.azkaban.install.dir","").getValue(); - - - - - - - - - - - - - - -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/ApplicationMapper.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/ApplicationMapper.java deleted file mode 100644 index a77f71b4f..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/ApplicationMapper.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.dao; - - -import com.webank.wedatasphere.dss.application.entity.Application; - -import java.util.List; - -/** - * Created by chaogefeng on 2019/10/10. - */ -public interface ApplicationMapper { - List listApplicationNames(); - - List listApplications(); - - Application getApplication(String appName); - - Application getApplicationbyNodeType(String nodeType); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/DSSApplicationUserMapper.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/DSSApplicationUserMapper.java deleted file mode 100644 index f2f59e06b..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/DSSApplicationUserMapper.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.dao; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; - -/** - * Created by chaogefeng on 2019/10/11. - */ -public interface DSSApplicationUserMapper { - DSSUser getUserByName(String username); - - void registerDssUser(DSSUser userDb); - - void updateUserFirstLogin(Long userId); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/LinkisUserMapper.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/LinkisUserMapper.java deleted file mode 100644 index b6d8a1048..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/LinkisUserMapper.java +++ /dev/null @@ -1,12 +0,0 @@ -package com.webank.wedatasphere.dss.application.dao; - -import com.webank.wedatasphere.dss.application.entity.LinkisUser; - -/** - * Created by chaogefeng on 2019/11/29. - */ -public interface LinkisUserMapper { - LinkisUser getUserByName(String username); - - void registerLinkisUser(LinkisUser userDb); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/ApplicationMapper.xml b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/ApplicationMapper.xml deleted file mode 100644 index 604cf39bf..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/ApplicationMapper.xml +++ /dev/null @@ -1,46 +0,0 @@ - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/DSSApplicationUserMapper.xml b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/DSSApplicationUserMapper.xml deleted file mode 100644 index 040cb3eb0..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/DSSApplicationUserMapper.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - - - - - - id,`username`,`name`,`is_first_login` - - - - - - INSERT INTO dss_user() - VALUES (#{id},#{username},#{name},#{isFirstLogin}) - - - - UPDATE FROM dss_user SET is_first_login = FALSE WHERE id = #{userId} - - - \ No newline at end of file diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/LinkisUserMapper.xml b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/LinkisUserMapper.xml deleted file mode 100644 index be55ca136..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/dao/impl/LinkisUserMapper.xml +++ /dev/null @@ -1,34 +0,0 @@ - - - - - - - - - - - - INSERT INTO linkis_user VALUES (#{id},#{email},#{userName},#{password},#{admin},#{active},#{name},#{description}, - #{department},#{avatar},#{createTime},#{createBy},#{updateTime},#{updateBy},#{isFirstLogin}) - - - - \ No newline at end of file diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/Application.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/Application.java deleted file mode 100644 index 80654846a..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/Application.java +++ /dev/null @@ -1,135 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.entity; - -/** - * Created by chaogefeng on 2019/10/10. - * “一个应用”的抽象，比如等等 - */ -public class Application { - private Integer id; - private String name; - private String url; - private Boolean isUserNeedInit; - private Integer level; - private String userInitUrl; - private Boolean existsProjectService;//有些应用可能没有project这么一个概念 - private String projectUrl; - private String enhanceJson; - private Boolean ifIframe; - private String homepageUrl; - private String redirectUrl; - - public Integer getId() { - return id; - } - - public void setId(Integer id) { - this.id = id; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getUrl() { - return url; - } - - public void setUrl(String url) { - this.url = url; - } - - public Boolean getUserNeedInit() { - return isUserNeedInit; - } - - public void setUserNeedInit(Boolean userNeedInit) { - isUserNeedInit = userNeedInit; - } - - public Integer getLevel() { - return level; - } - - public void setLevel(Integer level) { - this.level = level; - } - - public String getUserInitUrl() { - return userInitUrl; - } - - public void setUserInitUrl(String userInitUrl) { - this.userInitUrl = userInitUrl; - } - - public Boolean getExistsProjectService() { - return existsProjectService; - } - - public void setExistsProjectService(Boolean existsProjectService) { - this.existsProjectService = existsProjectService; - } - - public String getProjectUrl() { - return projectUrl; - } - - public void setProjectUrl(String projectUrl) { - this.projectUrl = projectUrl; - } - - public String getEnhanceJson() { - return enhanceJson; - } - - public void setEnhanceJson(String enhanceJson) { - this.enhanceJson = enhanceJson; - } - - public Boolean getIfIframe() { - return ifIframe; - } - - public String getHomepageUrl() { - return homepageUrl; - } - - public void setHomepageUrl(String homepageUrl) { - this.homepageUrl = homepageUrl; - } - - public String getRedirectUrl() { - return redirectUrl; - } - - public void setRedirectUrl(String redirectUrl) { - this.redirectUrl = redirectUrl; - } - - public void setIfIframe(Boolean ifIframe) { - this.ifIframe = ifIframe; - } - - -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/DSSUser.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/DSSUser.java deleted file mode 100644 index 73dd36aca..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/DSSUser.java +++ /dev/null @@ -1,68 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.entity; - -/** - * Created by chaogefeng on 2019/10/11. - */ -public class DSSUser { - private Long id; - private String username; - private String name; - private Boolean isFirstLogin; - private boolean isSuperUser = false; - - public boolean getIsSuperUser() { - return isSuperUser; - } - public void setIsSuperUser(boolean isSuperUser) { - this.isSuperUser = isSuperUser; - } - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getUsername() { - return username; - } - - public void setUsername(String username) { - this.username = username; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public Boolean getFirstLogin() { - return isFirstLogin; - } - - public void setFirstLogin(Boolean firstLogin) { - isFirstLogin = firstLogin; - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/DSSUserVO.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/DSSUserVO.java deleted file mode 100644 index 7c229c530..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/DSSUserVO.java +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.entity; - - - -/** - * Created by chaogefeng on 2019/10/11. - */ -public class DSSUserVO { - private DSSUser basic; - - public DSSUser getBasic() { - return basic; - } - - public void setBasic(DSSUser basic) { - this.basic = basic; - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/LinkisUser.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/LinkisUser.java deleted file mode 100644 index 240f38ce2..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/LinkisUser.java +++ /dev/null @@ -1,144 +0,0 @@ -package com.webank.wedatasphere.dss.application.entity; - -import java.util.Date; - -/** - * Created by chaogefeng on 2019/11/29. - */ -public class LinkisUser { - private Long id; - private String email; - private String userName; - private String password; - private Boolean admin; - private Boolean active; - private String name; - private String description; - private String department; - private String avatar; - private Date createTime; - private Long createBy; - private Date updateTime; - private Long updateBy; - private Boolean isFirstLogin; - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getEmail() { - return email; - } - - public void setEmail(String email) { - this.email = email; - } - - public String getUserName() { - return userName; - } - - public void setUserName(String userName) { - this.userName = userName; - } - - public String getPassword() { - return password; - } - - public void setPassword(String password) { - this.password = password; - } - - public Boolean getAdmin() { - return admin; - } - - public void setAdmin(Boolean admin) { - this.admin = admin; - } - - public Boolean getActive() { - return active; - } - - public void setActive(Boolean active) { - this.active = active; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public String getDepartment() { - return department; - } - - public void setDepartment(String department) { - this.department = department; - } - - public String getAvatar() { - return avatar; - } - - public void setAvatar(String avatar) { - this.avatar = avatar; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Long getCreateBy() { - return createBy; - } - - public void setCreateBy(Long createBy) { - this.createBy = createBy; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Long getUpdateBy() { - return updateBy; - } - - public void setUpdateBy(Long updateBy) { - this.updateBy = updateBy; - } - - public Boolean getFirstLogin() { - return isFirstLogin; - } - - public void setFirstLogin(Boolean firstLogin) { - isFirstLogin = firstLogin; - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/UserInitResult.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/UserInitResult.java deleted file mode 100644 index 87660e796..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/UserInitResult.java +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.entity; - -/** - * Created by chaogefeng on 2019/10/10. - */ -public class UserInitResult { - private Long id; - private Integer applicationId; - private String result; - private Long userId; - private Boolean isInitSuccess; - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public Integer getApplicationId() { - return applicationId; - } - - public void setApplicationId(Integer applicationId) { - this.applicationId = applicationId; - } - - public String getResult() { - return result; - } - - public void setResult(String result) { - this.result = result; - } - - public Long getUserId() { - return userId; - } - - public void setUserId(Long userId) { - this.userId = userId; - } - - public Boolean getInitSuccess() { - return isInitSuccess; - } - - public void setInitSuccess(Boolean initSuccess) { - isInitSuccess = initSuccess; - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/WorkSpacePath.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/WorkSpacePath.java deleted file mode 100644 index 4f2b25891..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/entity/WorkSpacePath.java +++ /dev/null @@ -1,59 +0,0 @@ -package com.webank.wedatasphere.dss.application.entity; - -public class WorkSpacePath { - private String workspaceRootPath; - private String hdfsRootPath; - private String resultRootPath; - private String schedulerPath; - private String userPath; - - public String getUserPath() { - return userPath; - } - - public void setUserPath(String userPath) { - this.userPath = userPath; - } - - - - public String getWorkspaceRootPath() { - return workspaceRootPath; - } - - public void setWorkspaceRootPath(String workspaceRootPath) { - this.workspaceRootPath = workspaceRootPath; - } - - public String getHdfsRootPath() { - return hdfsRootPath; - } - - public void setHdfsRootPath(String hdfsRootPath) { - this.hdfsRootPath = hdfsRootPath; - } - - public String getResultRootPath() { - return resultRootPath; - } - - public void setResultRootPath(String resultRootPath) { - this.resultRootPath = resultRootPath; - } - - public String getSchedulerPath() { - return schedulerPath; - } - - public void setSchedulerPath(String schedulerPath) { - this.schedulerPath = schedulerPath; - } - - - - - - - - -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/ApplicationHandlerChain.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/ApplicationHandlerChain.java deleted file mode 100644 index eff4b9518..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/ApplicationHandlerChain.java +++ /dev/null @@ -1,46 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.handler; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import java.util.Arrays; - -/** - * Created by chaogefeng on 2019/10/14. - */ -@Component -public class ApplicationHandlerChain { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private Handler[] handlers; - - public void handle(String user){ - DSSUser dataworkisUser = new DSSUser(); - dataworkisUser.setUsername(user); - dataworkisUser.setName(user); - Arrays.stream(handlers).sorted((h1,h2)-> h1.getOrder() - h2.getOrder()).forEach(handler -> handler.handle(dataworkisUser)); - } - -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/Handler.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/Handler.java deleted file mode 100644 index eb8f26204..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/Handler.java +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.handler; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; - -/** - * Created by chaogefeng on 2019/10/15. - */ -public interface Handler extends Order { - void handle(DSSUser user); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/LinkisUserFirstLoginHandler.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/LinkisUserFirstLoginHandler.java deleted file mode 100644 index 1a83caa72..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/LinkisUserFirstLoginHandler.java +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package com.webank.wedatasphere.dss.application.handler; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import com.webank.wedatasphere.dss.application.entity.LinkisUser; -import com.webank.wedatasphere.dss.application.service.LinkisUserService; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import java.util.Date; - -/** - * Created by chaogefeng on 2019/11/29. - */ -@Component -public class LinkisUserFirstLoginHandler implements Handler{ - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private LinkisUserService linkisUserService; - - @Override - public int getOrder() { - return -1; - } - - @Override - public void handle(DSSUser user) { - logger.info("LinkisUserFirstLoginHandler:"); - synchronized (user.getUsername().intern()){ - LinkisUser userDb = linkisUserService.getUserByName(user.getUsername()); - if(userDb == null){ - logger.info("User first enter linkis, insert table linkis_user and dss_user"); - userDb = new LinkisUser(); - userDb.setUserName(user.getUsername()); - userDb.setName(user.getName()); - userDb.setFirstLogin(true); - userDb.setCreateTime(new Date()); - userDb.setUpdateTime(new Date()); - linkisUserService.registerLinkisAndDssUser(userDb); - } - user.setId(userDb.getId()); - } - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/Order.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/Order.java deleted file mode 100644 index 30fb482c9..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/Order.java +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.handler; - -/** - * Created by chaogefeng on 2019/9/30. - */ -public interface Order { - - int getOrder(); - -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/ProjectUserInitHandler.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/ProjectUserInitHandler.java deleted file mode 100644 index 38f06b10a..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/ProjectUserInitHandler.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.handler; - - -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.stereotype.Component; - -/** - * Created by chaogefeng on 2019/10/15. - */ -@Component -public class ProjectUserInitHandler implements Handler { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Override - public void handle(DSSUser user) { - logger.info("ProjectUserInitHandler:"); - } - - @Override - public int getOrder() { - return 1; - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/UserFirstLoginHandler.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/UserFirstLoginHandler.java deleted file mode 100644 index f902c5ba3..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/handler/UserFirstLoginHandler.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.handler; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import com.webank.wedatasphere.dss.application.service.DSSApplicationUserService; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -/** - * Created by chaogefeng on 2019/10/15. - */ -@Component -public class UserFirstLoginHandler implements Handler { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private DSSApplicationUserService dssApplicationUserService; - - @Override - public int getOrder() { - return 0; - } - - @Override - public void handle(DSSUser user) { - logger.info("UserFirstLoginHandler:"); - synchronized (user.getUsername().intern()){ - DSSUser userDb = dssApplicationUserService.getUserByName(user.getUsername()); - if(userDb == null){ - logger.info("User first enter dss, insert table dss_user"); - userDb = new DSSUser(); - userDb.setUsername(user.getUsername()); - userDb.setName(user.getName()); - userDb.setFirstLogin(true); - userDb.setId(user.getId()); - dssApplicationUserService.registerDssUser(userDb); - } - // TODO: 2019/11/29 update firstLogin - user = userDb; - } - - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/restful/ApplicationRestfulApi.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/restful/ApplicationRestfulApi.java deleted file mode 100644 index 10552b25f..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/restful/ApplicationRestfulApi.java +++ /dev/null @@ -1,104 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.restful; - -import com.webank.wedatasphere.dss.application.conf.ApplicationConf; -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import com.webank.wedatasphere.dss.application.entity.DSSUserVO; -import com.webank.wedatasphere.dss.application.entity.WorkSpacePath; -import com.webank.wedatasphere.dss.application.handler.ApplicationHandlerChain; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import com.webank.wedatasphere.dss.application.service.DSSApplicationUserService; -import com.webank.wedatasphere.dss.application.util.ApplicationUtils; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.Consumes; -import javax.ws.rs.GET; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; - -/** - * Created by chaogefeng on 2019/10/10. - */ -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class ApplicationRestfulApi { - - @Autowired - private ApplicationService applicationService; - @Autowired - private DSSApplicationUserService dataworkisUserService; - @Autowired - private ApplicationHandlerChain applicationHandlerChain; - - @GET - @Path("getBaseInfo") - public Response getBaseInfo(@Context HttpServletRequest req){ - String username = SecurityFilter.getLoginUsername(req); - applicationHandlerChain.handle(username); - List applicationList = applicationService.listApplications(); - for (Application application : applicationList) { - String redirectUrl = application.getRedirectUrl(); - if(redirectUrl != null) { - application.setHomepageUrl(ApplicationUtils.redirectUrlFormat(redirectUrl,application.getHomepageUrl())); - application.setProjectUrl(ApplicationUtils.redirectUrlFormat(redirectUrl,application.getProjectUrl())); - } - } - DSSUser dssUser = dataworkisUserService.getUserByName(username); - DSSUserVO dataworkisUserVO = new DSSUserVO(); - String superUserName = ApplicationConf.SUPER_USER_NAME; - if(username.equals(superUserName)){ - dssUser.setIsSuperUser(true); - }else{ - dssUser.setIsSuperUser(false); - } - - dataworkisUserVO.setBasic(dssUser); - return Message.messageToResponse(Message.ok().data("applications",applicationList).data("userInfo",dataworkisUserVO)); - } - - - @GET - @Path("paths") - public Response getWorkSpace(@Context HttpServletRequest req) throws Exception { - WorkSpacePath workSpacePath = new WorkSpacePath(); - workSpacePath.setHdfsRootPath(ApplicationConf.HDFS_USER_ROOT_PATH); - workSpacePath.setResultRootPath(ApplicationConf.RESULT_SET_ROOT_PATH); - workSpacePath.setSchedulerPath(ApplicationConf.WDS_SCHEDULER_PATH); - workSpacePath.setWorkspaceRootPath(ApplicationConf.WORKSPACE_USER_ROOT_PATH); - workSpacePath.setUserPath(ApplicationConf.WDS_USER_PATH); - ArrayList> responses = ApplicationUtils.convertToMap(workSpacePath); - return Message.messageToResponse(Message.ok().data("paths",responses) - .data("dssInstallDir", ApplicationConf.DSS_INSTALL_DIR) - .data("azkakanDir", ApplicationConf.AZKABAN_INSTALL_DIR)); - - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/ApplicationService.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/ApplicationService.java deleted file mode 100644 index fa571e36f..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/ApplicationService.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.service; - - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.application.entity.Application; - -import java.util.List; - -/** - * Created by chaogefeng on 2019/10/10. - */ -public interface ApplicationService { - - List listApplicationNames(); - - List listApplications(); - - Application getApplication(String appName); - - AppJoint getAppjoint(String nodeType) throws AppJointErrorException; - - List listAppjoint() throws AppJointErrorException; - - Application getApplicationbyNodeType(String nodeType); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/DSSApplicationUserService.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/DSSApplicationUserService.java deleted file mode 100644 index a798dc435..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/DSSApplicationUserService.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.service; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; - -/** - * Created by chaogefeng on 2019/10/11. - */ -public interface DSSApplicationUserService { - - DSSUser getUserByName(String username); - - void registerDssUser(DSSUser userDb); - - void updateUserFirstLogin(Long id); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/LinkisUserService.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/LinkisUserService.java deleted file mode 100644 index 464de7849..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/LinkisUserService.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package com.webank.wedatasphere.dss.application.service; - -import com.webank.wedatasphere.dss.application.entity.LinkisUser; - -/** - * Created by chaogefeng on 2019/11/29. - */ -public interface LinkisUserService { - - LinkisUser getUserByName(String username); - - void registerLinkisAndDssUser(LinkisUser userDb); - - void registerDSSUser(LinkisUser userDb); -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/ApplicationServiceImpl.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/ApplicationServiceImpl.java deleted file mode 100644 index 271acac86..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/ApplicationServiceImpl.java +++ /dev/null @@ -1,110 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.service.impl; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.loader.AppJointLoader; -import com.webank.wedatasphere.dss.application.dao.ApplicationMapper; -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; -import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; -import org.apache.commons.lang.StringUtils; -import org.apache.htrace.fasterxml.jackson.databind.JavaType; -import org.apache.htrace.fasterxml.jackson.databind.ObjectMapper; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; - -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; - -/** - * Created by chaogefeng on 2019/10/10. - */ -@Service -public class ApplicationServiceImpl implements ApplicationService { - - private final Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private ApplicationMapper applicationMapper; - - @Override - public List listApplicationNames() { - return applicationMapper.listApplicationNames(); - } - - @Override - public List listApplications() { - return applicationMapper.listApplications(); - } - - @Override - public Application getApplication(String appName) { - return applicationMapper.getApplication(appName); - } - - @Override - public AppJoint getAppjoint(String nodeType) throws AppJointErrorException { - Application application; - if(nodeType.equals("schedulis")){ - application = getApplication(nodeType); - }else { - application = getApplicationbyNodeType(nodeType); - } - AppJoint appJoint = null; - try { - appJoint = loadAppjoint(application); - } catch (Exception e) { - logger.error("load appJoint failed:",e); - throw new AppJointErrorException(90000,"load appJoint failed:"+e.getMessage()); - } - return appJoint; - } - - @Override - public List listAppjoint()throws AppJointErrorException{ - List applications = listApplications(); - List appJoints = null; - try { - appJoints = applications.stream().filter(Application::getExistsProjectService).map(DSSExceptionUtils.map(this::loadAppjoint)).collect(Collectors.toList()); - }catch (DSSRuntimeException e){ - logger.error("load appJoint failed:",e); - throw new AppJointErrorException(90000,"load appJoint failed:"+e.getMessage()); - } - return appJoints; - } - - @Override - public Application getApplicationbyNodeType(String nodeType) { - return applicationMapper.getApplicationbyNodeType(nodeType); - } - - private AppJoint loadAppjoint(Application application) throws Exception { - String enhanceJson = application.getEnhanceJson(); - if(StringUtils.isBlank(enhanceJson)) enhanceJson = "{}"; - ObjectMapper om = new ObjectMapper(); - JavaType javaType = om.getTypeFactory().constructParametricType(Map.class,String.class,Object.class); - Map parmas = om.readValue(enhanceJson, javaType); - return AppJointLoader.getAppJointLoader().getAppJoint(application.getUrl(), application.getName(), parmas); - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/DSSApplicationUserServiceImpl.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/DSSApplicationUserServiceImpl.java deleted file mode 100644 index c8fae7a63..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/DSSApplicationUserServiceImpl.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.service.impl; - -import com.webank.wedatasphere.dss.application.dao.DSSApplicationUserMapper; -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import com.webank.wedatasphere.dss.application.service.DSSApplicationUserService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; - -/** - * Created by chaogefeng on 2019/10/11. - */ -@Service -public class DSSApplicationUserServiceImpl implements DSSApplicationUserService { - - @Autowired - private DSSApplicationUserMapper dssApplicationUserMapper; - - @Override - public DSSUser getUserByName(String username) { - return dssApplicationUserMapper.getUserByName(username); - } - - @Override - public void registerDssUser(DSSUser userDb) { - dssApplicationUserMapper.registerDssUser( userDb); - } - - @Override - public void updateUserFirstLogin(Long id) { - dssApplicationUserMapper.updateUserFirstLogin(id); - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/LinkisUserServiceImpl.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/LinkisUserServiceImpl.java deleted file mode 100644 index f40f286ec..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/service/impl/LinkisUserServiceImpl.java +++ /dev/null @@ -1,60 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package com.webank.wedatasphere.dss.application.service.impl; - -import com.webank.wedatasphere.dss.application.dao.DSSApplicationUserMapper; -import com.webank.wedatasphere.dss.application.dao.LinkisUserMapper; -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import com.webank.wedatasphere.dss.application.entity.LinkisUser; -import com.webank.wedatasphere.dss.application.service.LinkisUserService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -/** - * Created by chaogefeng on 2019/11/29. - */ -@Component -public class LinkisUserServiceImpl implements LinkisUserService { - - @Autowired - private LinkisUserMapper linkisUserMapper; - @Autowired - private DSSApplicationUserMapper dssApplicationUserMapper; - - @Override - public LinkisUser getUserByName(String username) { - return linkisUserMapper.getUserByName(username); - } - - - @Override - public void registerLinkisAndDssUser(LinkisUser userDb) { - // TODO: 2019/11/29 @transactional - linkisUserMapper.registerLinkisUser(userDb); - registerDSSUser(userDb); - } - - @Override - public void registerDSSUser(LinkisUser userDb) { - DSSUser dssUser = new DSSUser(); - dssUser.setId(userDb.getId()); - dssUser.setName(userDb.getName()); - dssUser.setUsername(userDb.getUserName()); - dssUser.setFirstLogin(userDb.getFirstLogin()); - dssApplicationUserMapper.registerDssUser(dssUser); - } -} diff --git a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/util/ApplicationUtils.java b/dss-application/src/main/java/com/webank/wedatasphere/dss/application/util/ApplicationUtils.java deleted file mode 100644 index af65bd322..000000000 --- a/dss-application/src/main/java/com/webank/wedatasphere/dss/application/util/ApplicationUtils.java +++ /dev/null @@ -1,85 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.application.util; - -import com.webank.wedatasphere.dss.application.entity.WorkSpacePath; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.UnsupportedEncodingException; -import java.lang.reflect.Field; -import java.net.URLEncoder; -import java.util.ArrayList; -import java.util.HashMap; - -/** - * Created by chaogefeng on 2019/11/20. - */ -public class ApplicationUtils { - - private static final Logger LOGGER = LoggerFactory.getLogger(ApplicationUtils.class); - - private static final String REDIRECT_FORMAT = "%s?redirect=%s&dssurl=${dssurl}&cookies=${cookies}"; - - private static String URLEndoder(String str){ - try { - return URLEncoder.encode(str,"utf-8"); - } catch (UnsupportedEncodingException e) { - LOGGER.warn("endoe failed:",e); - return str; - } - } - - public static String redirectUrlFormat(String redirectUrl,String url){ - return String.format(REDIRECT_FORMAT,redirectUrl,URLEndoder(url)); - } - - - public static ArrayList> convertToMap(Object obj) - throws Exception { - - ArrayList> mapList = new ArrayList(); - Field[] fields = obj.getClass().getDeclaredFields(); - for (int i = 0, len = fields.length; i < len; i++) { - String varName = fields[i].getName(); - boolean accessFlag = fields[i].isAccessible(); - fields[i].setAccessible(true); - HashMap map = new HashMap<>(); - Object o = fields[i].get(obj); - if (o != null){ - map.put("key",varName); - map.put("value",o.toString()); - mapList.add(map); - } - - fields[i].setAccessible(accessFlag); - } - - return mapList; - } - - public static void main(String[] args) throws Exception { -// System.out.println(redirectUrlFormat("http://127.0..0.1:8090/qualitis/api/v1/redirect","http://127.0..0.1:8090/#/projects/list?id={projectId}&flow=true")); - - WorkSpacePath workSpacePath = new WorkSpacePath(); - workSpacePath.setWorkspaceRootPath("/"); - System.out.println(convertToMap(workSpacePath)); - } - -} diff --git a/dss-apps/dss-apiservice-server/pom.xml b/dss-apps/dss-apiservice-server/pom.xml new file mode 100644 index 000000000..97f69346b --- /dev/null +++ b/dss-apps/dss-apiservice-server/pom.xml @@ -0,0 +1,420 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../pom.xml + + 4.0.0 + + dss-apiservice-server + + + + UTF-8 + 2.16 + 2.22.2 + 2.22.2 + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + asm + org.ow2.asm + + + hk2-api + org.glassfish.hk2 + + + jersey-common + org.glassfish.jersey.core + + + linkis-common + com.webank.wedatasphere.linkis + + + + + com.alibaba + fastjson + 1.2.70 + + + + org.glassfish.jersey.ext + jersey-bean-validation + ${jersey.version} + provided + + + javax.ws.rs-api + javax.ws.rs + + + hk2-locator + org.glassfish.hk2 + + + hk2-api + org.glassfish.hk2 + + + hibernate-validator + org.hibernate + + + jersey-server + org.glassfish.jersey.core + + + + + + com.webank.wedatasphere.linkis + linkis-bml-client + ${linkis.version} + + + commons-beanutils + commons-beanutils + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + com.webank.wedatasphere.linkis + linkis-computation-client + ${linkis.version} + + + commons-beanutils + commons-beanutils + + + linkis-common + com.webank.wedatasphere.linkis + + + + + org.postgresql + postgresql + 42.2.12 + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + + + linkis-common + com.webank.wedatasphere.linkis + + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + archaius-core + com.netflix.archaius + + + slf4j-api + org.slf4j + + + spring-cloud-starter + org.springframework.cloud + + + spring-web + org.springframework + + + + + com.zaxxer + HikariCP + 3.4.1 + + + slf4j-api + org.slf4j + + + + + + + + + + + hk2-api + org.glassfish.hk2 + 2.4.0-b34 + + + + + + + + + org.modelmapper + modelmapper + 0.7.5 + + + + io.jsonwebtoken + jjwt + 0.6.0 + + + jackson-databind + com.fasterxml.jackson.core + + + + + + org.hamcrest + hamcrest-all + 1.3 + test + + + + + com.h2database + h2 + 1.4.196 + test + + + + com.ninja-squad + DbSetup + 2.1.0 + test + + + org.springframework.boot + spring-boot-starter-test + ${spring.boot.version} + test + + + spring-boot-autoconfigure + org.springframework.boot + + + + + + junit + junit + 4.12 + test + + + org.junit.platform + junit-platform-launcher + 1.5.2 + test + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + xstream + com.thoughtworks.xstream + 1.4.11.1 + + + + + + + + maven-surefire-plugin + ${maven-surefire-plugin.version} + + + maven-failsafe-plugin + ${maven-failsafe-plugin.version} + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + + + + + + + + + + + + + ${project.artifactId}-${project.version} + + diff --git a/dss-apps/dss-apiservice-server/src/main/assembly/distribution.xml b/dss-apps/dss-apiservice-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..9882c22ec --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/assembly/distribution.xml @@ -0,0 +1,44 @@ + + + + dss-apiService-server + + dir + + true + dss-apiservice-server + + + + + + lib + true + true + false + true + true + + + + + + + diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/action/ResultSetDownloadAction.scala b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/action/ResultSetDownloadAction.scala new file mode 100644 index 000000000..856a85abf --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/action/ResultSetDownloadAction.scala @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.action + +import com.webank.wedatasphere.linkis.httpclient.request.{DownloadAction, GetAction} +import com.webank.wedatasphere.linkis.ujes.client.request.UJESJobAction + +import scala.tools.nsc.interpreter.InputStream + + + +class ResultSetDownloadAction extends GetAction with DownloadAction with UJESJobAction { + + private var inputStream: InputStream = _ + + override def write(inputStream: InputStream): Unit = this.inputStream = inputStream + + def getInputStream: InputStream = inputStream + + override def suffixURLs: Array[String] = Array("filesystem", "resultsetToExcel") +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceJob.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceJob.java new file mode 100644 index 000000000..d379ceb53 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceJob.java @@ -0,0 +1,51 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.bo; + +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; + + +public class ApiServiceJob { + + private String submitUser; + private String proxyUser; + private JobExecuteResult jobExecuteResult; + public String getSubmitUser() { + return submitUser; + } + + public void setSubmitUser(String submitUser) { + this.submitUser = submitUser; + } + + + + public String getProxyUser() { + return proxyUser; + } + + public void setProxyUser(String proxyUser) { + this.proxyUser = proxyUser; + } + public JobExecuteResult getJobExecuteResult() { + return jobExecuteResult; + } + + public void setJobExecuteResult(JobExecuteResult jobExecuteResult) { + this.jobExecuteResult = jobExecuteResult; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceQuery.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceQuery.java new file mode 100644 index 000000000..1f6035b55 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceQuery.java @@ -0,0 +1,122 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.bo; + + + +public class ApiServiceQuery { + + /** + * 服务名称 + */ + private String name; + + /** + * 标签 + */ + private String tag; + + /** + * 状态 + */ + private Integer status; + + /** + * api创建者 + */ + private String creator; + + private int currentPage; + + private int pageSize; + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + private int workspaceId; + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + private String userName; + + public ApiServiceQuery(String userName,String name, String tag, Integer status, String creator) { + this.userName= userName; + this.name = name; + this.tag = tag; + this.status = status; + this.creator = creator; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getTag() { + return tag; + } + + public void setTag(String tag) { + this.tag = tag; + } + + public Integer getStatus() { + return status; + } + + public void setStatus(Integer status) { + this.status = status; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public int getCurrentPage() { + return currentPage; + } + + public void setCurrentPage(int currentPage) { + this.currentPage = currentPage; + } + + public int getPageSize() { + return pageSize; + } + + public void setPageSize(int pageSize) { + this.pageSize = pageSize; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceToken.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceToken.java new file mode 100644 index 000000000..fb2b1b031 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/ApiServiceToken.java @@ -0,0 +1,66 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.bo; + + + +import java.util.Date; + + + +public class ApiServiceToken { + + private String applyUser; + private Date applyTime; + private Long apiServiceId; + private String publisher; + + public String getPublisher() { + return publisher; + } + + public void setPublisher(String publisher) { + this.publisher = publisher; + } + + public Long getApiServiceId() { + return apiServiceId; + } + + public void setApiServiceId(Long apiServiceId) { + this.apiServiceId = apiServiceId; + } + + public Date getApplyTime() { + return applyTime; + } + + public void setApplyTime(Date applyTime) { + this.applyTime = applyTime; + } + + public String getApplyUser() { + return applyUser; + } + + public void setApplyUser(String applyUser) { + this.applyUser = applyUser; + } + + + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/DirFileTree.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/DirFileTree.java new file mode 100644 index 000000000..30005e251 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/DirFileTree.java @@ -0,0 +1,79 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.bo; + +import java.util.HashMap; +import java.util.List; + + +public class DirFileTree { + + private String name; + private String path; + private HashMap properties; + private List children; + private Boolean isLeaf = false; + private String parentPath; + + public Boolean getIsLeaf() { + return isLeaf; + } + + public void setIsLeaf(Boolean isLeaf) { + this.isLeaf = isLeaf; + } + + public String getParentPath() { + return parentPath; + } + + public void setParentPath(String parentPath) { + this.parentPath = parentPath; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getPath() { + return path; + } + + public void setPath(String path) { + this.path = path; + } + + public HashMap getProperties() { + return properties; + } + + public void setProperties(HashMap properties) { + this.properties = properties; + } + + public List getChildren() { + return children; + } + + public void setChildren(List children) { + this.children = children; + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/LinkisExecuteResult.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/LinkisExecuteResult.java new file mode 100644 index 000000000..2314e9bfd --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/LinkisExecuteResult.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.bo; + + +public class LinkisExecuteResult { + + public LinkisExecuteResult(String taskId,String execId){ + this.taskId = taskId; + this.execId = execId; + } + + public String getTaskId() { + return taskId; + } + + public void setTaskId(String taskId) { + this.taskId = taskId; + } + + private String taskId; + + public String getExecId() { + return execId; + } + + public void setExecId(String execId) { + this.execId = execId; + } + + private String execId; +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/QueryRequest.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/QueryRequest.java new file mode 100644 index 000000000..87bec95d0 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/QueryRequest.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.bo; + +import java.util.Map; + + +public class QueryRequest { + /** + * 模块名，也是登录用户名、密码 + */ + private String moduleName; + + /** + * 调用参数 + */ + private Map params; + + public Map getParams() { + return params; + } + + public void setParams(Map params) { + this.params = params; + } + + public String getModuleName() { + return moduleName; + } + + public void setModuleName(String moduleName) { + this.moduleName = moduleName; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/TokenQuery.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/TokenQuery.java new file mode 100644 index 000000000..6cede5247 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/TokenQuery.java @@ -0,0 +1,117 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.bo; + + +import java.util.Date; + +public class TokenQuery { + private Long apiId; + private String user; + private Integer status; + + private String startDate; + private String endDate; + + private int currentPage; + + private int pageSize; + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + private String creator; + + + public TokenQuery(Long apiId, String user, Integer status) { + this.apiId = apiId; + this.user = user; + this.status = status; + } + + public TokenQuery(Long apiId, String user, Integer status, String startDate, String endDate) { + this.apiId = apiId; + this.user = user; + this.status = status; + this.startDate = startDate; + this.endDate = endDate; + } + + public TokenQuery() { + } + + public String getStartDate() { + return startDate; + } + + public void setStartDate(String startDate) { + this.startDate = startDate; + } + + public String getEndDate() { + return endDate; + } + + public void setEndDate(String endDate) { + this.endDate = endDate; + } + + public int getCurrentPage() { + return currentPage; + } + + public void setCurrentPage(int currentPage) { + this.currentPage = currentPage; + } + + public int getPageSize() { + return pageSize; + } + + public void setPageSize(int pageSize) { + this.pageSize = pageSize; + } + + public Long getApiId() { + return apiId; + } + + public void setApiId(Long apiId) { + this.apiId = apiId; + } + + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + public Integer getStatus() { + return status; + } + + public void setStatus(Integer status) { + this.status = status; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/VariableString.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/VariableString.java new file mode 100644 index 000000000..af901365c --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/bo/VariableString.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.bo; + + +public class VariableString { + private String path; + + public VariableString(String unparsedPath) { + this.path = unparsedPath; + } + + public String getPath() { + return path; + } + + public void setPath(String path) { + this.path = path; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/config/ApiServiceConfiguration.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/config/ApiServiceConfiguration.java new file mode 100644 index 000000000..ee670b169 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/config/ApiServiceConfiguration.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.config; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public class ApiServiceConfiguration { + public final static CommonVars LINKIS_AUTHOR_USER_TOKEN = CommonVars.apply("wds.linkis.client.api.service.author.user.token","WS-AUTH"); + public final static CommonVars LINKIS_ADMIN_USER = CommonVars.apply("wds.linkis.client.api.service.adminuser","ws"); + + public final static CommonVars LINKIS_CONNECTION_TIMEOUT = CommonVars.apply("wds.linkis.flow.connection.timeout",30000); + public final static CommonVars LINKIS_API_VERSION = CommonVars.apply("wds.linkis.server.version","v1"); + + public final static CommonVars API_SERVICE_TOKEN_KEY = CommonVars.apply("wds.dss.api.service.token.key","ApiServiceToken"); + + + public final static CommonVars DSS_API_TOKEN_SECRET_ID = CommonVars.apply("wds.dss.api.service.secret", "DSSSECRETTEST001002"); + + public final static CommonVars LINKIS_JOB_REQUEST_STATUS_TIME = CommonVars.apply("wds.linkis.job.status.timeout",3000); + + public final static CommonVars LOG_ARRAY_LEN = CommonVars.apply("wds.linkis.log.array.len",4); + + public final static CommonVars RESULT_PRINT_SIZE = CommonVars.apply("wds.linkis.result.print.size",10); + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ApiCommonConstant.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ApiCommonConstant.java new file mode 100644 index 000000000..a91850d29 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ApiCommonConstant.java @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.constant; + + +public class ApiCommonConstant { + public static final String DEFAULT_APPROVAL_NO="0001"; + public static final Integer API_DELETE_STATUS=2; +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ParamType.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ParamType.java new file mode 100644 index 000000000..56691675c --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ParamType.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.constant; + + +public class ParamType { + public static final String number = "2"; + + public static final String array = "4"; + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ParamTypeEnum.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ParamTypeEnum.java new file mode 100644 index 000000000..239469eef --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/ParamTypeEnum.java @@ -0,0 +1,56 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.constant; + + +public enum ParamTypeEnum { + + STRING(1, "String"), + + NUMBER(2, "Number"), + + DATE(3, "Date"), + + ARRAY(4, "Array"); + + private Integer index; + private String name; + + ParamTypeEnum(Integer index, String name) { + this.index = index; + this.name = name; + } + + public static ParamTypeEnum getEnum(Integer index) { + if (index == null) { + return STRING; + } + for (ParamTypeEnum statusEnum : values()) { + if (statusEnum.getIndex().equals(index)) { + return statusEnum; + } + } + return STRING; + } + + public Integer getIndex() { + return this.index; + } + + public String getName() { + return this.name; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/RequireEnum.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/RequireEnum.java new file mode 100644 index 000000000..a96906f00 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/RequireEnum.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.constant; + + +public enum RequireEnum { + + YES(1, "是"), + + NO(0, "否"); + + private Integer index; + private String name; + + RequireEnum(Integer index, String name) { + this.index = index; + this.name = name; + } + + public static RequireEnum getEnum(Integer index) { + if (index == null) { + return NO; + } + for (RequireEnum statusEnum : values()) { + if (statusEnum.getIndex().equals(index)) { + return statusEnum; + } + } + return NO; + } + + public Integer getIndex() { + return this.index; + } + + public String getName() { + return this.name; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/SQLMetadataInfoCheckStatus.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/SQLMetadataInfoCheckStatus.java new file mode 100644 index 000000000..15de7e2e5 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/SQLMetadataInfoCheckStatus.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.constant; + + + + +public enum SQLMetadataInfoCheckStatus { + LEGAL(0, "该SQL查询合法"), ILLEGALSQL(1,"不支持无库表信息的SQL查询"); + + int index; + String desc; + + SQLMetadataInfoCheckStatus(int index, String desc) { + this.index = index; + this.desc = desc; + } + + public String getDescByIndex(int index) { + for (SQLMetadataInfoCheckStatus item : values()) { + if (item.getIndex() == index) { + return item.getDesc(); + } + } + return null; + } + + public int getIndex() { + return index; + } + + public String getDesc() { + return desc; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/SaveTokenEnum.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/SaveTokenEnum.java new file mode 100644 index 000000000..8c6b5380c --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/SaveTokenEnum.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.constant; + + +public enum SaveTokenEnum { + SUCCESS(1, "success"), FAILED(2, "failed"), DUPLICATE(3, "duplicate"); + + int index; + String value; + SaveTokenEnum(int index, String value) { + this.index = index; + this.value = value; + } + + public int getIndex() { + return index; + } + + public void setIndex(int index) { + this.index = index; + } + + public String getValue() { + return value; + } + + public void setValue(String value) { + this.value = value; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/StatusEnum.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/StatusEnum.java new file mode 100644 index 000000000..c56cd6008 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/constant/StatusEnum.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.constant; + + +public enum StatusEnum { + + DISABLE(0, "停止"), + + ENABLE(1, "运行中"); + + private Integer index; + private String name; + + StatusEnum(Integer index, String name) { + this.index = index; + this.name = name; + } + + public static StatusEnum getEnum(Integer index) { + if (index == null) { + return null; + } + for (StatusEnum statusEnum : values()) { + if (statusEnum.getIndex().equals(index)) { + return statusEnum; + } + } + return null; + } + + public Integer getIndex() { + return this.index; + } + + public String getName() { + return this.name; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceAccessDao.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceAccessDao.java new file mode 100644 index 000000000..2b8c1b15d --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceAccessDao.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.dao; + +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiAccessVo; + + +public interface ApiServiceAccessDao { + void addAccessRecord(ApiAccessVo apiAccessVo); + + ApiAccessVo queryByVersionId(Long versionId); + + ApiAccessVo queryByApiId(Long apiServiceId); +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceDao.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceDao.java new file mode 100644 index 000000000..591bbf1a8 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceDao.java @@ -0,0 +1,81 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.dao; + + +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceQuery; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiServiceVo; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface ApiServiceDao { + + /** + * Insert + */ + void insert(ApiServiceVo apiServiceVo); + + /** + * Update + */ + void update(ApiServiceVo apiServiceVo); + + + void updateToTarget(ApiServiceVo apiServiceVo); + + /** + * query + */ + List query(ApiServiceQuery query); + + /** + * query + */ + + List queryByScriptPath(@Param("scriptPath") String scriptPath); + + /** + * query + */ + ApiServiceVo queryByPath(String path); + + /** + * query api path count + */ + Integer queryCountByPath(@Param("scriptPath") String scriptPath, @Param("path") String path); + + /** + * query api name + */ + Integer queryCountByName(String name); + + Integer enableApi(@Param("id") Long id); + + Integer disableApi(@Param("id") Long id); + + Integer deleteApi(@Param("id") Long id); + + List queryAllTags(@Param("userName")String userName,@Param("workspaceId")Integer workspaceId); + + ApiServiceVo queryById(@Param("id") Long id); + + List queryByWorkspaceId(@Param("workspaceId") Integer workspaceId, @Param("userName") String userName); + + Integer updateApiServiceComment(@Param("id") Long id, @Param("comment") String comment); +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceParamDao.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceParamDao.java new file mode 100644 index 000000000..918b7f40c --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceParamDao.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.dao; + + +import com.webank.wedatasphere.dss.apiservice.core.vo.ParamVo; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface ApiServiceParamDao { + + /** + * insert + * + * @param param + * @return + */ + int insert(ParamVo param); + + /** + * Update + * + * @param param + */ + void update(ParamVo param); + + /** + * query + */ +// List queryByConfigIdAndVersion(@Param("configId") Long configId, @Param("version") String version); + + List queryByVersionId(@Param("apiVersionId") Long apiVersionId); + + /** + * query a line by id + * @param paramId + * @return paramVo + * */ + int queryById(@Param("paramId") Long paramId); + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceTokenManagerDao.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceTokenManagerDao.java new file mode 100644 index 000000000..eea29f3a6 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceTokenManagerDao.java @@ -0,0 +1,115 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.dao; + +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; +import com.webank.wedatasphere.dss.apiservice.core.bo.TokenQuery; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface ApiServiceTokenManagerDao { + + /** + * insert a token + * */ + void insert(TokenManagerVo tokenManagerVo); + + /** + * insert batch token + * */ + void insertList(List tokenManagerVos); + + + /** + * 多条记录需要去重返回最新授权的 + * @param userName + * @return + */ + List queryByApplyUser(String userName); + + + List queryByApplyUserAndServiceId(@Param("userName") String userName, @Param("serviceId") Long serviceId); + + List queryByApplyUserAndVersionId(@Param("userName") String userName, @Param("apiVersionId") Long apiVersionId); + + + + /** + * query + * */ + List query(TokenQuery tokenQuery); + + /** + * query token by token id + * */ + TokenManagerVo queryTokenById(@Param("id") Long tokenId); + + /** + * query token by api service id + * */ + List queryTokenByApiServiceId(@Param("api_id") Long apiServiceId); + + /** + * query token avoid submit again + * */ + int queryApprovalNo(@Param("approvalNo") String approvalNo); + + /** + * query tokens according to status + * */ + List queryTokenByStatus(@Param("status") Integer status); + + /** + * disable token status + * */ + void disableTokenStatus(@Param("id") Long id); + + + /** + * disable token status + * */ + void disableTokenStatusByVersionId(@Param("apiVersionId") Long apiVersionId); + + + /** + * disable token status + * */ + void disableTokenStatusByApiId(@Param("apiId") Long apiId); + + /** + * enable token status + * */ + void enableTokenStatus(@Param("id") Long id); + + void enableTokenStatusByVersionId(@Param("apiVersionId") Long apiVersionId); + + void enableTokenStatusByApiId(@Param("apiId") Long apiId); + + + /** + * delete token + * */ + void deleteTokenById(@Param("id") Long tokenId); + + /** + * update token status + * */ +// void updateTokenStatus(@Param(id) Long tokenId, String status); + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceVersionDao.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceVersionDao.java new file mode 100644 index 000000000..521efd41f --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/ApiServiceVersionDao.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.dao; + + +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiVersionVo; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface ApiServiceVersionDao { + + /** + * inster apiversionVo + * */ + void insert(ApiVersionVo apiVersionVo); + + /** + * query version + * */ + ApiVersionVo queryApiVersionByVersionId(@Param("id") Long apiVersionId); + + List queryApiVersionByApiServiceId(@Param("api_id") Long apiServiceId); + + /** + * update api version + * */ + void updateApiVersionStatusById(@Param("id") Long apiVersionId, @Param("status") int status); + + void updateApiVersionStatus(ApiVersionVo apiVersionVo); + + /** + * update all api version status by service id + * */ + void updateAllApiVersionStatusByApiServiceId(@Param("api_id") Long apiServiceId, @Param("status") int status); + + void disableApiVersionStatusByApiId(@Param("api_id") Long apiServiceId); + + /** + * delete api version + * */ + void deleteApiVersionById(@Param("id") Long apiVersionId); +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceAccessMapper.xml b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceAccessMapper.xml new file mode 100644 index 000000000..ecda53d98 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceAccessMapper.xml @@ -0,0 +1,66 @@ + + + + + + + + + + + + + + + + + + `login_user`,`api_name`,`api_id`,`api_version_id`, `api_publisher`, `access_time`, `execute_user` + + + + `id`,`login_user`,`api_name`,`api_id`,`api_version_id`, `api_publisher`, `access_time`, `execute_user` + + + + + + + + + + + + + + + + diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceApiVersionMapper.xml b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceApiVersionMapper.xml new file mode 100644 index 000000000..48460f34b --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceApiVersionMapper.xml @@ -0,0 +1,109 @@ + + + + + + + + + + + + + + + + + + + + + + `api_id`,`version`,`bml_resource_id`,`bml_version`,`source`,`creator`,`create_time`,`status`, + `metadata_info`, `auth_id` + + + + `id`,`api_id`,`version`,`bml_resource_id`,`bml_version`,`source`,`creator`,`create_time`,`status`, + `metadata_info`, `auth_id` + + + + + + + + + + + + + + UPDATE `dss_apiservice_api_version` + SET `status` = #{status} + WHERE id = #{id} + + + + UPDATE `dss_apiservice_api_version` + SET `status` = 0 + WHERE + api_id = #{api_id} + + + + UPDATE `dss_apiservice_api_version` + SET `status` = #{status} + WHERE id = #{id} + + + + UPDATE `dss_apiservice_api_version` + SET `status` = #{status} + WHERE api_id = #{api_id} + + + + + DELETE FROM + `dss_apiservice_api_version` + WHERE + id = #{id} + + + + \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceMapper.xml b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceMapper.xml new file mode 100644 index 000000000..34d6ab302 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceMapper.xml @@ -0,0 +1,297 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + `name`,`alias_name`, `path`,`protocol`,`method`,`tag`, `scope`,`description`,`status`,`type`,`run_type`,`create_time`, + `modify_time`,`creator`,`modifier`,`script_path`,`workspaceID`,`api_comment` + + + + `id`, `alias_name`, `name`,`path`,`protocol`,`method`, `tag`, `scope`,`description`,`status`,`type`,`run_type`,`create_time`, + `modify_time`,`creator`,`modifier`,`script_path`,`workspaceID`,`api_comment` + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + `id` = #{id} + + + + + + + + + + + + + + + + + + + + + + + + + + + + `id` = #{id} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceParamMapper.xml b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceParamMapper.xml new file mode 100644 index 000000000..e1b449032 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceParamMapper.xml @@ -0,0 +1,98 @@ + + + + + + + + + + + + + + + + + + + `api_version_id`,`name`,`type`,`required`,`description`, `display_name`, `default_value`, `details` + + + + `id`, `api_version_id`, `name`,`type`,`required`,`description`, `display_name`, `default_value`, `details` + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + `id` = #{id} + + + + + + + + + + + + + + + + diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceTokenManagerMapper.xml b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceTokenManagerMapper.xml new file mode 100644 index 000000000..1ee6c7399 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/dao/mapper/ApiServiceTokenManagerMapper.xml @@ -0,0 +1,245 @@ + + + + + + + + + + + + + + + + + + + + + + + + `api_id`,`api_version_id`,`publisher`,`user`,`apply_time`,`duration`, + `reason`,`ip_whitelist`,`status`,`caller`,`access_limit`,`token`,`apply_source` + + + + `id`, `api_version_id`,`api_id`,`publisher`,`user`,`apply_time`,`duration`, + `reason`,`ip_whitelist`,`status`,`caller`,`access_limit`,`token`,`apply_source` + + + + `id`, `api_version_id`,`api_id`,`publisher`,`user`,`apply_time`,`duration`, + `reason`,`ip_whitelist`,`status`,`caller`,`access_limit`,`apply_source` + + + + + + + + + + + + + + + (#{item.apiId},#{item.apiVersionId}, #{item.publisher}, #{item.user}, #{item.applyTime}, + #{item.duration}, #{item.reason}, #{item.ipWhitelist}, #{item.status}, + #{item.caller}, #{item.accessLimit}, #{item.token}, #{item.applySource}) + + + + + + + + + + + + + + + + + + + + + + + + + + + UPDATE `dss_apiservice_token_manager` + SET `status` = 1 + WHERE `id` = #{id} + + + + UPDATE `dss_apiservice_token_manager` + SET `status` = 0 + WHERE `id` = #{id} + + + + UPDATE `dss_apiservice_token_manager` + SET `status` = 0 + WHERE `api_version_id` = #{apiVersionId} + + + + + UPDATE `dss_apiservice_token_manager` + SET `status` = 1 + WHERE `api_version_id` = #{apiVersionId} + + + + + + + UPDATE `dss_apiservice_token_manager` + SET `status` = 0 + WHERE `api_id` = #{apiId} + + + + + UPDATE `dss_apiservice_token_manager` + SET `status` = 1 + WHERE `api_id` = #{apiId} + + + + DELETE FROM + `dss_apiservice_token_manager` + WHERE + id = #{id} + + + \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiExecuteException.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiExecuteException.java new file mode 100644 index 000000000..6023284d7 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiExecuteException.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class ApiExecuteException extends ErrorException { + + public ApiExecuteException(int errCode, String desc) { + super(errCode, desc); + } + + public ApiExecuteException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } + } diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceQueryException.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceQueryException.java new file mode 100644 index 000000000..e415c3e84 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceQueryException.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class ApiServiceQueryException extends ErrorException { + + public ApiServiceQueryException(int errCode, String desc) { + super(errCode, desc); + } + + public ApiServiceQueryException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceRuntimeException.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceRuntimeException.java new file mode 100644 index 000000000..4ac983f1c --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceRuntimeException.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.exception; + + +public class ApiServiceRuntimeException extends RuntimeException { + + public ApiServiceRuntimeException(String message) { + super(message); + } + + public ApiServiceRuntimeException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceTokenException.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceTokenException.java new file mode 100644 index 000000000..23805a7fa --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/ApiServiceTokenException.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class ApiServiceTokenException extends ErrorException { + + public ApiServiceTokenException(int errCode, String desc) { + super(errCode, desc); + } + + public ApiServiceTokenException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } + } diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/AssertException.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/AssertException.java new file mode 100644 index 000000000..346b93722 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/AssertException.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.exception; + + +public class AssertException extends ApiServiceRuntimeException { + + public AssertException(String message) { + super(message); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/BeanValidationExceptionMapper.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/BeanValidationExceptionMapper.java new file mode 100644 index 000000000..5304b01ee --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/exception/BeanValidationExceptionMapper.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.exception; + +import com.webank.wedatasphere.linkis.server.Message; + +import javax.validation.ConstraintViolationException; +import javax.validation.ValidationException; +import javax.ws.rs.core.Response; +import javax.ws.rs.ext.ExceptionMapper; +import javax.ws.rs.ext.Provider; + + +@Provider +public class BeanValidationExceptionMapper implements ExceptionMapper { + @Override + public Response toResponse(ValidationException exception) { + StringBuilder stringBuilder = new StringBuilder(); + ((ConstraintViolationException)exception) + .getConstraintViolations().forEach(constraintViolation -> stringBuilder.append(constraintViolation.getMessage()).append(";")); + Message message = Message.error("Bean validation error[实例校验出错], detail:" + stringBuilder.toString()); + return Message.messageToResponse(message); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/ApiServiceExecuteJob.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/ApiServiceExecuteJob.java new file mode 100644 index 000000000..86bd42db2 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/ApiServiceExecuteJob.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.execute; + +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; + +import java.util.Map; + + +public interface ApiServiceExecuteJob { + String getCode(); + + void setCode(String code); + + String getEngineType(); + + void setEngineType(String engineType); + + String getRunType(); + + void setRunType(String runType); + + String getUser(); + + void setUser(String user); + + String getJobName(); + + Map getParams(); + + void setParams(Map params); + + Map getRuntimeParams(); + void setRuntimeParams(Map runtimeParams); + + JobExecuteResult getJobExecuteResult(); + + void setJobExecuteResult(JobExecuteResult jobExecuteResult); + + Map getJobProps(); + + void setJobProps(Map jobProps); + + String getScriptPath(); + void setScriptePath(String path); +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/DefaultApiServiceJob.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/DefaultApiServiceJob.java new file mode 100644 index 000000000..daf6e6b87 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/DefaultApiServiceJob.java @@ -0,0 +1,141 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.execute; + +import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; + +import java.util.Map; + + +public class DefaultApiServiceJob implements ApiServiceExecuteJob { + private Map jobProps; + private String user; + private Map source; + private Map variables; + + private Map configuration; + + private String code; + + private String engineType; + + private String runType; + + private Map params; + + private Map runtimeParams; + + private JobExecuteResult jobExecuteResult; + + private String scriptPath; + + @Override + public String getCode() { + return this.code; + } + + @Override + public void setCode(String code) { + this.code = code; + } + + @Override + public String getEngineType() { + return engineType; + } + + @Override + public void setEngineType(String engineType) { + this.engineType = engineType; + } + + @Override + public String getRunType() { + return runType; + } + + @Override + public void setRunType(String runType) { + this.runType = runType; + } + + @Override + public String getUser() { + return user; + } + + @Override + public void setUser(String user){ + this.user =user; + } + + @Override + public String getJobName() { + return null; + } + + @Override + public Map getParams() { + return params; + } + + @Override + public void setParams(Map params) { + this.params = params; + } + + @Override + public Map getRuntimeParams() { + return runtimeParams; + } + + @Override + public void setRuntimeParams(Map runtimeParams) { + this.runtimeParams = runtimeParams; + } + + @Override + public JobExecuteResult getJobExecuteResult() { + return this.jobExecuteResult; + } + + @Override + public void setJobExecuteResult(JobExecuteResult jobExecuteResult) { + this.jobExecuteResult = jobExecuteResult; + } + + @Override + public Map getJobProps() { + return jobProps; + } + + @Override + public void setJobProps(Map jobProps) { + this.jobProps = jobProps; + } + + @Override + public String getScriptPath() { + return scriptPath; + } + + @Override + public void setScriptePath(String path) { + this.scriptPath = path; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/ExecuteCodeHelper.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/ExecuteCodeHelper.java new file mode 100644 index 000000000..0cab36b23 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/ExecuteCodeHelper.java @@ -0,0 +1,216 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.execute; + + +import com.webank.wedatasphere.dss.apiservice.core.action.ResultSetDownloadAction; +import com.webank.wedatasphere.dss.apiservice.core.config.ApiServiceConfiguration; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiExecuteException; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import com.webank.wedatasphere.linkis.httpclient.response.Result; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.request.ResultSetAction; +import com.webank.wedatasphere.linkis.ujes.client.request.ResultSetListAction; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; +import com.webank.wedatasphere.linkis.ujes.client.response.JobInfoResult; +import com.webank.wedatasphere.linkis.ujes.client.response.JobLogResult; +import com.webank.wedatasphere.linkis.ujes.client.response.ResultSetListResult; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.InputStream; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Map; + +public class ExecuteCodeHelper { + + private static final Logger LOGGER = LoggerFactory.getLogger(ExecuteCodeHelper.class); + + private static final String RELEASE_SCALA_TEMPLATE = "import org.apache.spark.sql.DataFrame\n" + + "val sql = %s\n" + + "val df = sqlContext.sql(sql)\n" + + "show(df)\n"; + + + private static final String SCALA_MARK = "\"\"\""; + + private static final String EXECUTE_SCALA_TEMPLATE = "import org.apache.spark.sql.DataFrame\n" + + "val executeCode = %s\n" + + "val df = sqlContext.sql(executeCode)\n" + + "show(df)\n"; + + + + public static String packageCodeToRelease(String executeCode){ + String retStr = String.format(RELEASE_SCALA_TEMPLATE, SCALA_MARK + executeCode + SCALA_MARK); + LOGGER.info("release scala code is {}", retStr); + return retStr; + } + + public static String packageCodeToExecute(String executeCode, String metaDataInfo){ + String retStr = String.format(EXECUTE_SCALA_TEMPLATE, SCALA_MARK + executeCode + SCALA_MARK); + LOGGER.info("execute scala code is {}", retStr); + return retStr; + } + + + public static void waitForComplete(ApiServiceExecuteJob job,UJESClient client) throws Exception { + JobInfoResult jobInfo = client.getJobInfo(job.getJobExecuteResult()); + while (!jobInfo.isCompleted()) { + LOGGER.info("Update Progress info:" + getProgress(job,client)); + LOGGER.info("<----linkis log ---->"); + Utils.sleepQuietly(ApiServiceConfiguration.LINKIS_JOB_REQUEST_STATUS_TIME.getValue(job.getJobProps())); + jobInfo = client.getJobInfo(job.getJobExecuteResult()); + } + if (!jobInfo.isSucceed()) { + throw new ApiExecuteException(90101, "Failed to execute Job: " + jobInfo.getTask().get("errDesc").toString()); + } + } + + + public static void cancel(ApiServiceExecuteJob job,UJESClient client) throws Exception { + client.kill(job.getJobExecuteResult()); + } + + + public static double getProgress(ApiServiceExecuteJob job,UJESClient client) { + return client.progress(job.getJobExecuteResult()).getProgress(); + } + + + public static Boolean isCompleted(ApiServiceExecuteJob job,UJESClient client) { + return client.getJobInfo(job.getJobExecuteResult()).isCompleted(); + } + + public static String getResult(ApiServiceExecuteJob job, int index, int maxSize,UJESClient client) { + String resultContent = null; + JobInfoResult jobInfo = client.getJobInfo(job.getJobExecuteResult()); + String[] resultSetList = jobInfo.getResultSetList(LinkisJobSubmit.getClient(job.getJobProps())); + if (resultSetList != null && resultSetList.length > 0) { + Object fileContent = client.resultSet(ResultSetAction.builder() + .setPath(resultSetList[index]) + .setUser(job.getJobExecuteResult().getUser()) + .setPageSize(maxSize).build()).getFileContent(); + if (fileContent instanceof ArrayList) { + ArrayList> resultSetRow = (ArrayList>) fileContent; + resultContent = StringUtils.join(resultSetRow.get(0), "\n"); + } else { + resultContent = fileContent.toString(); + } + } + return resultContent; + } + + + public static int getResultSize(ApiServiceExecuteJob job,UJESClient client) { + JobInfoResult jobInfo = client.getJobInfo(job.getJobExecuteResult()); + if (jobInfo.isSucceed()) { + String[] resultSetList = jobInfo.getResultSetList(LinkisJobSubmit.getClient(job.getJobProps())); + if (resultSetList != null && resultSetList.length > 0) { + return resultSetList.length; + } + } + return 0; + } + + + public static String getLog(ApiServiceExecuteJob job, UJESClient client) { + + JobLogResult jobLogResult = client + .log(job.getJobExecuteResult(), + 0, + 50); + + ArrayList logArray = jobLogResult.getLog(); + + if (logArray != null && logArray.size() + >= ApiServiceConfiguration.LOG_ARRAY_LEN.getValue() + && StringUtils.isNotEmpty(logArray.get(3))) { + return logArray.get(3); + } + return null; + } + + public static String getResultList(JobExecuteResult executeResult,UJESClient client, String path) { + ResultSetListResult resultList = (ResultSetListResult) client.executeUJESJob(ResultSetListAction.builder() + .setUser(executeResult.getUser()).setPath(path).build()); + return resultList.getResponseBody(); + } + + + + public static String getResultContent(String user, String path, int maxSize,UJESClient client) { + + String fileContent = client.resultSet(ResultSetAction.builder() + .setPath(path) + .setUser(user) + .setPageSize(maxSize).build()).getResponseBody(); + + return fileContent; + } + + public static InputStream downloadResultSet(String user, + String path, + String charset, + String outputFileType, + String csvSeperator, + String outputFileName, + String sheetName, + String nullValue, + UJESClient client) { + + ResultSetDownloadAction resultSetDownloadAction = new ResultSetDownloadAction(); + resultSetDownloadAction.setUser(user); + resultSetDownloadAction.setParameter("path",path); + resultSetDownloadAction.setParameter("charset",charset); + resultSetDownloadAction.setParameter("outputFileType",outputFileType); + resultSetDownloadAction.setParameter("csvSeperator",csvSeperator); + resultSetDownloadAction.setParameter("outputFileName",outputFileName); + resultSetDownloadAction.setParameter("sheetName",sheetName); + resultSetDownloadAction.setParameter("nullValue",nullValue); + client.executeUJESJob(resultSetDownloadAction); + return resultSetDownloadAction.getInputStream(); + } + + + + + + + + public static Map getTaskInfoById(JobExecuteResult jobExecuteResult, UJESClient client) { + + Map taskInfo = (Map)client.getJobInfo(jobExecuteResult).getTask(); + + return taskInfo; + } + + + + + + + + + + + + + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/LinkisJobSubmit.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/LinkisJobSubmit.java new file mode 100644 index 000000000..a24691670 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/execute/LinkisJobSubmit.java @@ -0,0 +1,100 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.execute; + +import com.webank.wedatasphere.dss.apiservice.core.config.ApiServiceConfiguration; +import com.webank.wedatasphere.linkis.common.conf.Configuration; +import com.webank.wedatasphere.linkis.httpclient.dws.authentication.TokenAuthenticationStrategy; +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfig; +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfigBuilder; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.UJESClientImpl; +import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; + +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.TimeUnit; + + +public class LinkisJobSubmit { + + + + public static UJESClient getClient(Map props) { + + UJESClient client = getUJESClient( + Configuration.GATEWAY_URL().getValue(props), + ApiServiceConfiguration.LINKIS_ADMIN_USER.getValue(props), + ApiServiceConfiguration.LINKIS_AUTHOR_USER_TOKEN.getValue(props), + props); + + return client; + } + + + public static UJESClient getUJESClient(String url, String user, String token, Map jobProps){ + return new UJESClientImpl(getClientConfig(url,user,token, jobProps)); + } + + public static DWSClientConfig getClientConfig(String url, String user, String token, Map jobProps){ + DWSClientConfig clientConfig = ((DWSClientConfigBuilder) (DWSClientConfigBuilder.newBuilder() + .addServerUrl(url) + .connectionTimeout(ApiServiceConfiguration.LINKIS_CONNECTION_TIMEOUT.getValue(jobProps)) + .discoveryEnabled(false).discoveryFrequency(1, TimeUnit.MINUTES) + .loadbalancerEnabled(true) + .maxConnectionSize(5) + .retryEnabled(false).readTimeout(ApiServiceConfiguration.LINKIS_CONNECTION_TIMEOUT.getValue(jobProps)) + .setAuthenticationStrategy(new TokenAuthenticationStrategy()) + .setAuthTokenKey(user).setAuthTokenValue(token))) + .setDWSVersion(ApiServiceConfiguration.LINKIS_API_VERSION.getValue(jobProps)).build(); + return clientConfig; + } + + + + public static JobExecuteResult execute(ApiServiceExecuteJob job,UJESClient client) { + Map source = new HashMap<>(); + source.put("DSS-ApiService",job.getScriptPath()); + JobExecuteAction.Builder builder = JobExecuteAction.builder().setCreator("IDE") + .addExecuteCode(job.getCode()) + .setEngineTypeStr(job.getEngineType()) + .setRunTypeStr(job.getRunType()) + .setUser(job.getUser()) + .setParams(job.getParams()) + .setVariableMap(job.getRuntimeParams()) + .setSource(source); + JobExecuteAction jobAction = builder.build(); + JobExecuteResult res = client.execute(jobAction); + return res; + } + public static JobExecuteResult execute(ApiServiceExecuteJob job,UJESClient client, String creator) { + Map source = new HashMap<>(); + source.put("DSS-ApiService",job.getScriptPath()); + JobExecuteAction.Builder builder = JobExecuteAction.builder().setCreator(creator) + .addExecuteCode(job.getCode()) + .setEngineTypeStr(job.getEngineType()) + .setRunTypeStr(job.getRunType()) + .setUser(job.getUser()) + .setParams(job.getParams()) + .setVariableMap(job.getRuntimeParams()) + .setSource(source); + JobExecuteAction jobAction = builder.build(); + JobExecuteResult res = client.execute(jobAction); + return res; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/jdbc/DatasourceService.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/jdbc/DatasourceService.java new file mode 100644 index 000000000..0b246ae86 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/jdbc/DatasourceService.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.jdbc; + +import com.zaxxer.hikari.HikariConfig; +import com.zaxxer.hikari.HikariDataSource; +import org.springframework.context.annotation.Configuration; +import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate; + +import javax.sql.DataSource; +import java.util.HashMap; +import java.util.Map; + + +@Configuration +public class DatasourceService { + private static final Map CLIENT_MAP = new HashMap<>(); + + + private DataSource getDatasource(String url, String username, String password) { + HikariConfig hikariConfig = new HikariConfig(); + hikariConfig.setJdbcUrl(url); + hikariConfig.setUsername(username); + hikariConfig.setPassword(password); + hikariConfig.setMaximumPoolSize(100); + hikariConfig.setMinimumIdle(10); + return new HikariDataSource(hikariConfig); + } + + + public NamedParameterJdbcTemplate getNamedParameterJdbcTemplate(String url, + String username, String password) { + String key = genKey(url, username); + NamedParameterJdbcTemplate jdbcTemplate = CLIENT_MAP.get(key); + + if (jdbcTemplate == null) { + synchronized (this) { + jdbcTemplate = CLIENT_MAP.get(key); + if (jdbcTemplate == null) { + jdbcTemplate = new NamedParameterJdbcTemplate(getDatasource(url, username, password)); + CLIENT_MAP.put(key, jdbcTemplate); + } + } + } + + return jdbcTemplate; + } + + private String genKey(String jdbcUrl, String username) { + return String.join("-", jdbcUrl, username); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/jdbc/JdbcUtil.scala b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/jdbc/JdbcUtil.scala new file mode 100644 index 000000000..00ce6a1b8 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/jdbc/JdbcUtil.scala @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.jdbc + +import java.util + +import com.webank.wedatasphere.linkis.common.exception.WarnException +import com.webank.wedatasphere.linkis.common.utils.Logging +//import com.webank.wedatasphere.linkis.datasourcemanager.common.protocol.{DsInfoQueryRequest, DsInfoResponse} +import com.webank.wedatasphere.linkis.rpc.Sender + +object JdbcUtil extends Logging { + +// val sender : Sender = Sender.getSender("dsm-server") +// def getDatasourceInfo(params : util.Map[String, Any]) : (String, String, String) = { +// val datasourceId = params.get("configuration").asInstanceOf[util.Map[String, Any]] +// .getOrDefault("datasource", new util.HashMap[String, Any]()) +// .asInstanceOf[util.Map[String, Any]].get("datasourceId") +// logger.info(s"begin to get datasource info from dsm, datasourceId: ${datasourceId}") +// if (datasourceId != null) { +// val ds = sender.ask(DsInfoQueryRequest(String.valueOf(datasourceId), "BDP")) match { +// case r: DsInfoResponse => r +// case warn: WarnException => throw warn +// } +// logger.info(s"get datasource info result: ${ds}") +// if (ds.status) { +// val url = ds.params.get("jdbc.url").asInstanceOf[String] +// val userName = ds.params.get("jdbc.username").asInstanceOf[String] +// val password = ds.params.get("jdbc.password").asInstanceOf[String] +// logger.info(s"get from dsm: url: ${url}, username: ${userName}, password: ${password}") +// return (url, userName, password) +// } +// } +// +// ("", "", "") +// } + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceCoreRestfulApi.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceCoreRestfulApi.java new file mode 100644 index 000000000..24f5680f8 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceCoreRestfulApi.java @@ -0,0 +1,538 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.restful; + +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceQuery; +import com.webank.wedatasphere.dss.apiservice.core.util.ApiUtils; +import com.webank.wedatasphere.dss.apiservice.core.util.AssertUtil; +import com.webank.wedatasphere.dss.apiservice.core.vo.*; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiServiceQueryService; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiService; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.collections.MapUtils; +import org.apache.commons.lang.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.validation.ConstraintViolation; +import javax.validation.ConstraintViolationException; +import javax.validation.Validator; +import javax.validation.groups.Default; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.*; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + + +@Path("/dss/apiservice") +@Consumes(MediaType.APPLICATION_JSON) +@Produces(MediaType.APPLICATION_JSON) +@Component +public class ApiServiceCoreRestfulApi { + + private static final Logger LOG = LoggerFactory.getLogger(ApiServiceCoreRestfulApi.class); + + @Autowired + private ApiService apiService; + + @Autowired + private ApiServiceQueryService apiServiceQueryService; + + @Autowired + private Validator beanValidator; + + private static final Pattern WRITABLE_PATTERN = Pattern.compile("^\\s*(insert|update|delete|drop|alter|create).*", Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + + @POST + @Path("/api") + public Response insert(ApiServiceVo apiService, @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + + if (apiService.getWorkspaceId() == null){ + apiService.setWorkspaceId(180L); + } + + if (StringUtils.isBlank(apiService.getAliasName())) { + return Message.error("'api service alias name' is missing[缺少中文名]"); + } + + if (StringUtils.isBlank(apiService.getScriptPath())) { + return Message.error("'api service script path' is missing[缺少脚本路径]"); + } + if (StringUtils.isBlank(apiService.getContent())) { + return Message.error("'api service script content' is missing[缺少脚本内容]"); + } + + if (null == apiService.getWorkspaceId()) { + return Message.error("'api service workspaceId ' is missing[缺少工作空间ID]"); + } + if (apiService.getContent().contains(";")) { + if(!apiService.getContent().toLowerCase().startsWith("use ")) { + return Message.error("'api service script content exists semicolon[脚本内容包含分号]"); + } + } + +// check data change script + if (WRITABLE_PATTERN.matcher(apiService.getContent()).matches()) { + return Message.error("'api service script content' only supports query[脚本内容只支持查询语句]"); + } + + Map metadata = apiService.getMetadata(); + if (apiService.getScriptPath().endsWith(".jdbc")) { + if (MapUtils.isEmpty(metadata)) { + return Message.error("'api service metadata' is missing[请选择数据源]"); + } + + Map configuration = (Map) metadata.get("configuration"); + if (MapUtils.isEmpty(configuration)) { + return Message.error("'api service metadata.configuration' is missing[请选择数据源]"); + } + + Map datasource = (Map) configuration.get("datasource"); + if (MapUtils.isEmpty(datasource)) { + return Message.error("'api service metadata.configuration.datasource' is missing[请选择数据源]"); + } + } + + String userName = SecurityFilter.getLoginUsername(req); + Set> result = beanValidator.validate(apiService, Default.class); + if (result.size() > 0) { + throw new ConstraintViolationException(result); + } + + ApprovalVo approvalVo = apiService.getApprovalVo(); + +// if (StringUtils.isBlank(approvalVo.getApprovalName())) { +// return Message.error("'approvalName' is missing[缺少审批单名字]"); +// } + + if (StringUtils.isBlank(approvalVo.getApplyUser())) { + return Message.error("'applyUser' is missing[缺少申请用户名字]"); + } + + apiService.setCreator(userName); + apiService.setModifier(userName); + this.apiService.save(apiService); + return Message.ok().data("insert_id", apiService.getId()).data("approval_no",approvalVo.getApprovalNo()); + }, "/apiservice/api", "Fail to insert service api[新增服务api失败]"); + } + + @POST + @Path("/create") + public Response create(ApiServiceVo apiService, @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + + if (apiService.getWorkspaceId() == null){ + apiService.setWorkspaceId(180L); + } + + if (StringUtils.isBlank(apiService.getAliasName())) { + return Message.error("'api service alias name' is missing[缺少中文名]"); + } + + if (StringUtils.isBlank(apiService.getScriptPath())) { + return Message.error("'api service script path' is missing[缺少脚本路径]"); + } + if (StringUtils.isBlank(apiService.getContent())) { + return Message.error("'api service script content' is missing[缺少脚本内容]"); + } + + if (null == apiService.getWorkspaceId()) { + return Message.error("'api service workspaceId ' is missing[缺少工作空间ID]"); + } + if (apiService.getContent().contains(";")) { + if(!apiService.getContent().toLowerCase().startsWith("use ")) { + return Message.error("'api service script content exists semicolon[脚本内容包含分号]"); + } + } + +// check data change script + if (WRITABLE_PATTERN.matcher(apiService.getContent()).matches()) { + return Message.error("'api service script content' only supports query[脚本内容只支持查询语句]"); + } + + Map metadata = apiService.getMetadata(); + if (apiService.getScriptPath().endsWith(".jdbc")) { + if (MapUtils.isEmpty(metadata)) { + return Message.error("'api service metadata' is missing[请选择数据源]"); + } + + Map configuration = (Map) metadata.get("configuration"); + if (MapUtils.isEmpty(configuration)) { + return Message.error("'api service metadata.configuration' is missing[请选择数据源]"); + } + + Map datasource = (Map) configuration.get("datasource"); + if (MapUtils.isEmpty(datasource)) { + return Message.error("'api service metadata.configuration.datasource' is missing[请选择数据源]"); + } + } + + String userName = SecurityFilter.getLoginUsername(req); + Set> result = beanValidator.validate(apiService, Default.class); + if (result.size() > 0) { + throw new ConstraintViolationException(result); + } + + ApprovalVo approvalVo = apiService.getApprovalVo(); + + if (StringUtils.isBlank(approvalVo.getApprovalName())) { + return Message.error("'approvalName' is missing[缺少审批单名字]"); + } + + if (StringUtils.isBlank(approvalVo.getApplyUser())) { + return Message.error("'applyUser' is missing[缺少申请用户名字]"); + } + + apiService.setCreator(userName); + apiService.setModifier(userName); + this.apiService.saveByApp(apiService); + return Message.ok().data("insert_id", apiService.getId()).data("approval_no",approvalVo.getApprovalNo()); + }, "/apiservice/api", "Fail to insert service api[新增服务api失败]"); + } + + @PUT + @Path("/api/{api_service_version_id}") + public Response update(ApiServiceVo apiService, + @PathParam("api_service_version_id") Long apiServiceVersionId, + @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + + + if (StringUtils.isBlank(apiService.getScriptPath())) { + return Message.error("'api service script path' is missing[缺少脚本路径]"); + } + if(apiServiceVersionId !=0) { + if (StringUtils.isBlank(apiService.getPath())) { + return Message.error("'api service api path' is missing[缺少api路径]"); + } + } + if (StringUtils.isBlank(apiService.getContent())) { + return Message.error("'api service script content' is missing[缺少脚本内容]"); + } + + if (null == apiService.getWorkspaceId()) { + return Message.error("'api service workspaceId ' is missing[缺少工作空间ID]"); + } + + if (null == apiService.getTargetServiceId()) { + return Message.error("'api service update to target service id ' is missing[缺少更新目标服务ID]"); + } + + if (apiService.getContent().contains(";")) { + return Message.error("'api service script content exists semicolon[脚本内容包含分号]"); + } + + ApprovalVo approvalVo = apiService.getApprovalVo(); + +// if (StringUtils.isBlank(approvalVo.getApprovalName())) { +// return Message.error("'approvalName' is missing[缺少审批单名字]"); +// } + + if (StringUtils.isBlank(approvalVo.getApplyUser())) { + return Message.error("'applyUser' is missing[缺少申请用户名字]"); + } +// if (StringUtils.isBlank(apiService.getResourceId())) { +// return Message.error("'api service resourceId' is missing[缺少bml resourceId]"); +// } + +// check data change script + if (WRITABLE_PATTERN.matcher(apiService.getContent()).matches()) { + return Message.error("'api service script content' only supports query[脚本内容只支持查询语句]"); + } + + Map metadata = apiService.getMetadata(); + if (apiService.getScriptPath().endsWith(".jdbc")) { + if (MapUtils.isEmpty(metadata)) { + return Message.error("'api service metadata' is missing[请选择数据源]"); + } + + Map configuration = (Map) metadata.get("configuration"); + if (MapUtils.isEmpty(configuration)) { + return Message.error("'api service metadata.configuration' is missing[请选择数据源]"); + } + + Map datasource = (Map) configuration.get("datasource"); + if (MapUtils.isEmpty(datasource)) { + return Message.error("'api service metadata.configuration.datasource' is missing[请选择数据源]"); + } + } + + String userName = SecurityFilter.getLoginUsername(req); +// Bean validation + Set> result = beanValidator.validate(apiService, Default.class); + if (result.size() > 0) { + throw new ConstraintViolationException(result); + } + apiService.setLatestVersionId(apiServiceVersionId); + apiService.setModifier(userName); + apiService.setModifyTime(Calendar.getInstance().getTime()); + this.apiService.update(apiService); + return Message.ok().data("update_id", apiServiceVersionId); + }, "/apiservice/api/" + apiServiceVersionId, "Fail to update service api[更新服务api失败]"); + } + + + + + @GET + @Path("/search") + public Response query(@QueryParam("name") String name, + @QueryParam("tag") String tag, + @QueryParam("status") Integer status, + @QueryParam("creator") String creator, + @QueryParam("workspaceId") Integer workspaceId, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + + return ApiUtils.doAndResponse(() -> { + if (null == workspaceId) { + return Message.error("'api service search workspaceId' is missing[缺少工作空间Id]"); + } + ApiServiceQuery query = new ApiServiceQuery(userName,name, tag, status, creator); + query.setWorkspaceId(workspaceId); + if(!this.apiService.checkUserWorkspace(userName,workspaceId) ){ + return Message.error("'api service search workspaceId' is wrong[该用户不属于该工作空间Id]"); + } + List queryList = apiService.query(query); + return Message.ok().data("query_list", queryList); + }, "/apiservice/search", "Fail to query page of service api[查询服务api失败]"); + } + + + @GET + @Path("/getUserServices") + public Response getUserServices(@QueryParam("workspaceId") Integer workspaceId, + @Context HttpServletRequest req){ + String userName = SecurityFilter.getLoginUsername(req); + return ApiUtils.doAndResponse(() -> { + if(!this.apiService.checkUserWorkspace(userName,workspaceId) ){ + return Message.error("'api service getUserServices workspaceId' is wrong[该用户不属于该工作空间Id]"); + } + List apiServiceList = apiService.queryByWorkspaceId(workspaceId,userName); + return Message.ok().data("query_list", apiServiceList); + }, "/apiservice/getUserServices", "Fail to query page of user service api[查询用户服务api列表失败]"); + } + + + + @GET + @Path("/tags") + public Response query( @Context HttpServletRequest req,@QueryParam("workspaceId") Integer workspaceId) { + String userName = SecurityFilter.getLoginUsername(req); + return ApiUtils.doAndResponse(() -> { + + List tags= apiService.queryAllTags(userName,workspaceId); + return Message.ok().data("tags", tags); + }, "/apiservice/tags", "Fail to query page of service tag[查询服务tag失败]"); + } + + + + + @GET + @Path("/query") + public Response queryByScriptPath(@QueryParam("scriptPath") String scriptPath, + @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (StringUtils.isBlank(scriptPath)) { + return Message.error("'api service scriptPath' is missing[缺少脚本路径]"); + } + ApiServiceVo apiServiceVo = apiService.queryByScriptPath(scriptPath); + if(null != apiServiceVo) { + if (!this.apiService.checkUserWorkspace(userName, apiServiceVo.getWorkspaceId().intValue())) { + return Message.error("'api service query workspaceId' is wrong[该用户不属于该工作空间Id]"); + } + + if (apiServiceVo.getCreator().equals(userName)) { + return Message.ok().data("result", apiServiceVo); + } else { + return Message.error("'api service belong to others' [该脚本路径数据服务不属于当前用户]"); + } + }else { + return Message.ok().data("result", apiServiceVo); + } + }, "/apiservice/query", "Fail to query page of service api[查询服务api失败]"); + } + + @GET + @Path("/queryById") + public Response queryById(@QueryParam("id") Long id, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return ApiUtils.doAndResponse(() -> { + if (id==null) { + return Message.error("'api service id' is missing[缺少服务ID]"); + } + ApiServiceVo apiServiceVo = apiService.queryById(id,userName); + AssertUtil.notNull(apiServiceVo,"未找到数据服务，有可能已经被删除"); + if(!this.apiService.checkUserWorkspace(userName,apiServiceVo.getWorkspaceId().intValue()) ){ + return Message.error("'api service queryById for workspaceId' is wrong[该用户不属于该工作空间Id]"); + } + return Message.ok().data("result", apiServiceVo); + }, "/apiservice/queryById", "Fail to query page of service api[查询服务api失败]"); + } + + @GET + @Path("/checkPath") + public Response checkPath(@QueryParam("scriptPath") String scriptPath, @QueryParam("path") String path) { + //需要跨用户查询 + return ApiUtils.doAndResponse(() -> { + if (StringUtils.isBlank(scriptPath)) { + return Message.error("'api service scriptPath' is missing[缺少api脚本路径]"); + } + if (StringUtils.isBlank(path)) { + return Message.error("'api service path' is missing[缺少api路径]"); + } + Integer apiCount = apiService.queryCountByPath(scriptPath, path); + return Message.ok().data("result", 0 > Integer.valueOf(0).compareTo(apiCount)); + }, "/apiservice/checkPath", "Fail to check path of service api[校验服务api路径失败]"); + } + + @GET + @Path("/checkName") + public Response checkName(@QueryParam("name") String name) { + //需要跨用户查询 + return ApiUtils.doAndResponse(() -> { + if (StringUtils.isBlank(name)) { + return Message.error("'api service name' is missing[缺少api名称]"); + } + Integer count = apiService.queryCountByName(name); + return Message.ok().data("result", count > 0); + }, "/apiservice/checkName", "Fail to check name of service api[校验服务api名称失败]"); + } + + @GET + @Path("/apiDisable") + public Response apiDisable(@QueryParam("id") Long id, + @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (null == id) { + return Message.error("'api service api id' is missing[缺少api id]"); + } + boolean resultFlag = apiService.disableApi(id,userName); + return Message.ok().data("result", resultFlag); + }, "/apiservice/apiDisable", "Fail to disable api[禁用api失败]"); + } + + @GET + @Path("/apiEnable") + public Response apiEnable(@QueryParam("id") Long id, + @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (null == id) { + return Message.error("'api service api id' is missing[缺少api id]"); + } + boolean resultFlag = apiService.enableApi(id,userName); + return Message.ok().data("result", resultFlag); + }, "/apiservice/apiEnable", "Fail to enable api[启用api失败]"); + } + + @GET + @Path("/apiDelete") + public Response apiDelete(@QueryParam("id") Long id, + @Context HttpServletRequest req) { + //目前暂时不实际删除数据，只做不可见和不可用。 + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (null == id) { + return Message.error("'api service api id' is missing[缺少api id]"); + } + boolean resultFlag = apiService.deleteApi(id,userName); + return Message.ok().data("result", resultFlag); + }, "/apiservice/apiDelete", "Fail to delete api[删除api失败]"); + } + + @POST + @Path("/apiCommentUpdate") + public Response apiCommentUpdate(@Context HttpServletRequest req, JsonNode json) { + //目前暂时不实际删除数据，只做不可见和不可用。 + Long id = json.get("id").getLongValue(); + String comment=json.get("comment").getTextValue(); + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (null == id) { + return Message.error("'api service api id' is missing[缺少api id]"); + } + boolean resultFlag = apiService.updateComment(id,comment,userName); + return Message.ok().data("result", resultFlag); + }, "/apiservice/apiDelete", "Fail to delete api[删除api失败]"); + } + + + @GET + @Path("/apiParamQuery") + public Response apiParamQuery(@QueryParam("scriptPath") String scriptPath, + @QueryParam("versionId") Long versionId, + @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (StringUtils.isEmpty(scriptPath)) { + return Message.error("'api service api scriptPath' is missing[缺少api scriptPath]"); + } + if (null == versionId) { + return Message.error("'api service api version' is missing[缺少api 版本号]"); + } + List queryParamVoList = apiServiceQueryService.queryParamList(scriptPath, versionId); + return Message.ok().data("result", queryParamVoList); + }, "/apiservice/apiParamQuery", "Fail to query api info[查询api信息失败]"); + } + + @GET + @Path("/apiVersionQuery") + public Response apiVersionQuery(@QueryParam("serviceId") Long serviceId, + @Context HttpServletRequest req) { + return ApiUtils.doAndResponse(() -> { + String userName = SecurityFilter.getLoginUsername(req); + if (null == serviceId) { + return Message.error("'api service api serviceId' is missing[缺少api serviceId]"); + } + List apiVersionVoList = apiServiceQueryService.queryApiVersionById(serviceId) + .stream().filter(apiVersionVo -> apiVersionVo.getCreator().equals(userName)) + .collect(Collectors.toList()); + return Message.ok().data("result", apiVersionVoList); + }, "/apiservice/apiVersionQuery", "Fail to query api version[查询api版本失败]"); + } + + @GET + @Path("/apiContentQuery") + public Response apiContentQuery(@QueryParam("versionId") Long versionId, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return ApiUtils.doAndResponse(() -> { + if (null== versionId) { + return Message.error("'api service api versionId' is missing[缺少api versionId]"); + } + ApiServiceVo apiServiceVo = apiServiceQueryService.queryByVersionId(userName,versionId); + if(!this.apiService.checkUserWorkspace(userName,apiServiceVo.getWorkspaceId().intValue()) ){ + return Message.error("'api service apiContentQuery for workspaceId' is wrong[该用户不属于该工作空间Id]"); + } + return Message.ok().data("result", apiServiceVo); + }, "/apiservice/apiContentQuery", "Fail to query api Content[查询api版本内容失败]"); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceExecuteRestfulApi.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceExecuteRestfulApi.java new file mode 100644 index 000000000..fc5141c87 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceExecuteRestfulApi.java @@ -0,0 +1,316 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.restful; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JavaType; +import com.webank.wedatasphere.dss.apiservice.core.bo.*; +import com.webank.wedatasphere.dss.apiservice.core.config.ApiServiceConfiguration; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceQueryException; +import com.webank.wedatasphere.dss.apiservice.core.execute.ExecuteCodeHelper; +import com.webank.wedatasphere.dss.apiservice.core.execute.LinkisJobSubmit; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiServiceQueryService; +import com.webank.wedatasphere.dss.apiservice.core.token.JwtManager; +import com.webank.wedatasphere.dss.apiservice.core.util.ApiUtils; +import com.webank.wedatasphere.dss.apiservice.core.util.AssertUtil; +import com.webank.wedatasphere.dss.apiservice.core.vo.*; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; +import org.apache.commons.collections4.MapUtils; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; +import org.springframework.util.StringUtils; +import org.springframework.web.bind.annotation.RequestBody; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.io.IOException; +import java.io.InputStream; +import java.util.Enumeration; +import java.util.HashMap; +import java.util.Map; + + + +@Path("/dss/apiservice") +@Consumes(MediaType.APPLICATION_JSON) +@Produces(MediaType.APPLICATION_JSON) +@Component +public class ApiServiceExecuteRestfulApi { + public static final String XLSX_RESPONSE_CONTENT_TYPE = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"; + + private static final Logger logger = LoggerFactory.getLogger(ApiServiceExecuteRestfulApi.class); + @Autowired + private ApiServiceQueryService queryService; + + + private static final String SYS_COLUMN_PREFIX = "_"; + + private static final String requestBodyDemo = "{\"moduleName\":\"aladdin-demo\",\"params\":{\"param1\": \"value1\"}}"; + + + + @POST + @Path("/execute/{path:.*}") + public Response post(@PathParam("path") VariableString path, @RequestBody QueryRequest queryRequest, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return getResponse(userName,path.getPath(), queryRequest, HttpMethod.POST); + } + + @GET + @Path("/execute/{path:.*}") + public Response get(@PathParam("path") VariableString path, + @Context HttpServletRequest req) throws JsonProcessingException { + String userName = SecurityFilter.getLoginUsername(req); + + QueryRequest queryRequest = new QueryRequest(); + + + // 如果requestBody为空，尝试从url中获取参数 + if (MapUtils.isEmpty(queryRequest.getParams())) { + String paramSuffix = "params."; + Enumeration parameterNames = req.getParameterNames(); + Map params = new HashMap<>(); + while (parameterNames.hasMoreElements()) { + String name = parameterNames.nextElement(); + if (name.startsWith(paramSuffix)) { + params.put(name.substring(paramSuffix.length()), req.getParameter(name)); + } + } + queryRequest.setParams(params); + } + String paramsJsonStr = req.getParameter("params"); + JavaType javaType = BDPJettyServerHelper.jacksonJson().getTypeFactory().constructParametricType(Map.class,String.class,Object.class); + Map parmas = BDPJettyServerHelper.jacksonJson().readValue(paramsJsonStr, javaType); + queryRequest.setParams(parmas); + + if (StringUtils.isEmpty(queryRequest.getModuleName())) { + queryRequest.setModuleName(req.getParameter("moduleName")); + } + + return getResponse(userName,path.getPath(), queryRequest, HttpMethod.GET); + } + + @PUT + @Path("/execute/{path:.*}") + public Response put(@PathParam("path") VariableString path, @RequestBody QueryRequest queryRequest, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return getResponse(userName,path.getPath(), queryRequest, HttpMethod.PUT); + } + + @DELETE + @Path("/execute/{path:.*}") + public Response delete(@PathParam("path") VariableString path, @RequestBody QueryRequest queryRequest, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return getResponse(userName,path.getPath(), queryRequest, HttpMethod.DELETE); + } + + private void validParam(QueryRequest queryRequest) { + AssertUtil.notNull(queryRequest, "请求体不能为空，正确的格式:" + requestBodyDemo); + AssertUtil.notEmpty(queryRequest.getModuleName(), "moduleName不能为空，正确的格式:" + requestBodyDemo); + AssertUtil.notNull(queryRequest.getParams().get(ApiServiceConfiguration.API_SERVICE_TOKEN_KEY.getValue()),"请求token不能为空"); + } + + @GET + @Path("/getDirFileTrees") + public void getDirFileTrees(@Context HttpServletRequest req, @Context HttpServletResponse resp, + @QueryParam("path") String path, + @QueryParam("taskId") String taskId) throws IOException, ApiServiceQueryException { + String userName = SecurityFilter.getLoginUsername(req); + if (StringUtils.isEmpty(path)) { + throw new ApiServiceQueryException(80004, path); + } + String dirFileTree=""; + ApiServiceJob apiServiceJob = queryService.getJobByTaskId(taskId); + if(null != apiServiceJob && userName.equals(apiServiceJob.getSubmitUser())) { + JobExecuteResult jobExecuteResult = new JobExecuteResult(); + jobExecuteResult.setTaskID(taskId); + jobExecuteResult.setUser(apiServiceJob.getProxyUser()); + Map props = new HashMap<>(); + UJESClient client = LinkisJobSubmit.getClient(props); + + + dirFileTree = ExecuteCodeHelper.getResultList(jobExecuteResult, client, path); + }else{ + dirFileTree="当前用户不存在运行的TaskId: "+taskId; + } + resp.getWriter().println(dirFileTree); + resp.getWriter().flush(); + } + + + @GET + @Path("/openFile") + public void openFile(@Context HttpServletRequest req, + @QueryParam("path") String path, + @QueryParam("taskId") String taskId, + @DefaultValue("1") @QueryParam("page") Integer page, + @DefaultValue("5000") @QueryParam("pageSize") Integer pageSize, + @DefaultValue("utf-8") @QueryParam("charset") String charset, + @Context HttpServletResponse resp) throws IOException, ApiServiceQueryException { + String userName = SecurityFilter.getLoginUsername(req); + if (StringUtils.isEmpty(path)) { + throw new ApiServiceQueryException(80004, path); + } + if(StringUtils.isEmpty(taskId)){ + throw new ApiServiceQueryException(80005, "taskId is null"); + } + String fileContent=""; + ApiServiceJob apiServiceJob = queryService.getJobByTaskId(taskId); + if(null != apiServiceJob && userName.equals(apiServiceJob.getSubmitUser())) { + Map props = new HashMap<>(); + UJESClient client = LinkisJobSubmit.getClient(props); + fileContent = ExecuteCodeHelper.getResultContent(apiServiceJob.getProxyUser(), path, pageSize, client); + }else{ + fileContent="当前用户不存在运行的TaskId: "+taskId; + } + resp.getWriter().println(fileContent); + resp.getWriter().flush(); + } + + @GET + @Path("resultsetToExcel") + public void resultsetToExcel( + @Context HttpServletRequest req, + @Context HttpServletResponse resp, + @QueryParam("path") String path, + @QueryParam("taskId") String taskId, + @DefaultValue("utf-8") @QueryParam("charset") String charset, + @DefaultValue("csv") @QueryParam("outputFileType") String outputFileType, + @DefaultValue(",") @QueryParam("csvSeperator") String csvSeperator, + @DefaultValue("downloadResultset") @QueryParam("outputFileName") String outputFileName, + @DefaultValue("result") @QueryParam("sheetName") String sheetName, + @DefaultValue("NULL") @QueryParam("nullValue") String nullValue) throws ApiServiceQueryException, IOException { + + resp.addHeader("Content-Disposition", "attachment;filename=" + + new String(outputFileName.getBytes("UTF-8"), "ISO8859-1") + "." + outputFileType); + resp.setCharacterEncoding(charset); + + switch (outputFileType) { + case "csv": + resp.addHeader("Content-Type", "text/plain"); + break; + case "xlsx": + resp.addHeader("Content-Type", XLSX_RESPONSE_CONTENT_TYPE); + break; + default: + new ApiServiceQueryException(80015,"不支持的下载类型"); + + } + + String userName = SecurityFilter.getLoginUsername(req); + if (StringUtils.isEmpty(path)) { + throw new ApiServiceQueryException(80005, path); + } + InputStream inputStream; + ApiServiceJob apiServiceJob = queryService.getJobByTaskId(taskId); + if(null != apiServiceJob && userName.equals(apiServiceJob.getSubmitUser())) { + Map props = new HashMap<>(); + UJESClient client = LinkisJobSubmit.getClient(props); + inputStream = ExecuteCodeHelper.downloadResultSet(apiServiceJob.getProxyUser(), + path, + charset, + outputFileType, + csvSeperator, + outputFileName, + sheetName, + nullValue, + client); + } else{ + resp.getWriter().println("当前用户不存在运行的TaskId: "+taskId); + resp.getWriter().flush(); + return; + } + try { + IOUtils.copy(inputStream, resp.getOutputStream()); + resp.getOutputStream().flush(); + } finally { + IOUtils.closeQuietly(inputStream); + } + } + + @GET + @Path("/{id}/get") + public Response getTaskByID(@Context HttpServletRequest req, @PathParam("id") Long taskId) { + String username = SecurityFilter.getLoginUsername(req); + ApiServiceJob apiServiceJob = queryService.getJobByTaskId(taskId.toString()); + if (null != apiServiceJob && username.equals(apiServiceJob.getSubmitUser())) { + Map props = new HashMap<>(); + UJESClient client = LinkisJobSubmit.getClient(props); + JobExecuteResult jobExecuteResult = apiServiceJob.getJobExecuteResult(); + jobExecuteResult.setUser(apiServiceJob.getProxyUser()); + Map vo = ExecuteCodeHelper.getTaskInfoById(jobExecuteResult, client); + return Message.messageToResponse(Message.ok().data("task", vo)); + } else { + return Message.messageToResponse(Message.ok().data("task", null)); + } + } + + private Response getResponse(String user,String path, QueryRequest queryRequest, String httpMethod) { + return ApiUtils.doAndResponse(() -> { + + validParam(queryRequest); + String token = queryRequest.getParams().get(ApiServiceConfiguration.API_SERVICE_TOKEN_KEY.getValue()).toString(); + + MessageVo messageVo = null; + ApiServiceToken tokenDetail = null; + boolean isParseRight = true; + try { + tokenDetail = JwtManager.parseToken(token); + }catch (Exception e) { + isParseRight = false; + messageVo = new MessageVo().setData("token解析错误，该token无效！"); + } + if(false == isParseRight) { + return messageVo; + } + + if(tokenDetail.getApplyUser().equals(user)) { + + LinkisExecuteResult query = queryService.query("/" + path, + queryRequest.getParams() == null ? new HashMap<>() : queryRequest.getParams(), + queryRequest.getModuleName(), httpMethod,tokenDetail,user); + if(null == query) { + messageVo = new MessageVo().setMessage("用户任务执行出错，用户参数错误！").setStatus(1); + return messageVo; + } + + HashMap queryRes = new HashMap<>(); + queryRes.put("taskId",query.getTaskId()); + queryRes.put("execId",query.getExecId()); + messageVo = new MessageVo().setData(queryRes); + }else { + messageVo = new MessageVo().setData("Token is not correct"); + } + + return messageVo; + }); + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceTokenRestfulApi.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceTokenRestfulApi.java new file mode 100644 index 000000000..51704d2d8 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/restful/ApiServiceTokenRestfulApi.java @@ -0,0 +1,112 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.restful; + +import com.github.pagehelper.PageInfo; +import com.webank.wedatasphere.dss.apiservice.core.constant.SaveTokenEnum; +import com.webank.wedatasphere.dss.apiservice.core.token.TokenAuth; +import com.webank.wedatasphere.dss.apiservice.core.service.TokenQueryService; +import com.webank.wedatasphere.dss.apiservice.core.util.ApiUtils; +import com.webank.wedatasphere.dss.apiservice.core.util.DateUtil; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApprovalVo; +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; +import com.webank.wedatasphere.dss.apiservice.core.bo.TokenQuery; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.Calendar; +import java.util.Date; +import java.util.List; + + + +@Path("/dss/apiservice") +@Consumes(MediaType.APPLICATION_JSON) +@Produces(MediaType.APPLICATION_JSON) +@Component +public class ApiServiceTokenRestfulApi { + private static final Logger LOG = LoggerFactory.getLogger(ApiServiceTokenRestfulApi.class); + + @Autowired + TokenQueryService tokenQueryService; + + + + @Autowired + TokenAuth tokenAuth; + + + @GET + @Path("/tokenQuery") + public Response apiServiceTokenQuery(@QueryParam("apiId") Long apiId, + @QueryParam("user") String user, + @QueryParam("status") Integer status, + @QueryParam("startDate") String startDateStr, + @QueryParam("endDate") String endDateStr, + @QueryParam("currentPage") Integer currentPage, + @QueryParam("pageSize") Integer pageSize, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return ApiUtils.doAndResponse(() -> { + + TokenQuery query = null; + + if(StringUtils.isBlank(startDateStr) || StringUtils.isBlank(endDateStr)) { + query = new TokenQuery(apiId, user, status); + } else { + Long startDateSecond = Long.parseLong(startDateStr); + Long endDateSeconed = Long.parseLong(endDateStr); + String startDate = DateUtil.format(new Date(startDateSecond), DateUtil.FORMAT_LONG); + Date endDateReal = new Date(endDateSeconed); + Calendar cal = Calendar.getInstance(); + cal.setTime(endDateReal); + cal.add(Calendar.DATE, 1); + Date realDate = cal.getTime(); + String endDate = DateUtil.format(realDate, DateUtil.FORMAT_LONG); + + query = new TokenQuery(apiId, user, status, startDate, endDate); + + } + query.setCreator(userName); + query.setCurrentPage(null != currentPage ? currentPage : 1); + query.setPageSize(null != pageSize ? pageSize : 10); + PageInfo tokenList = tokenQueryService.query(query); + return Message.ok().data("queryList", tokenList.getList()).data("total", tokenList.getTotal()); + }, "/apiservice/tokenQuery", "Fail to query page of token[查询token信息失败]"); + } + + + @GET + @Path("/approvalRefresh") + public Response refresh(@QueryParam("approvalNo") String approvalNo, + @Context HttpServletRequest req) { + String userName = SecurityFilter.getLoginUsername(req); + return ApiUtils.doAndResponse(() ->{ + return Message.ok().data("approvalStatus", "success"); + }, "/apiservice/approvalRefresh/", "查询出错"); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/ApiService.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/ApiService.java new file mode 100644 index 000000000..8937efaeb --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/ApiService.java @@ -0,0 +1,98 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.service; + +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceQuery; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceQueryException; +import com.webank.wedatasphere.dss.apiservice.core.vo.*; + +import java.util.List; + + +public interface ApiService { + + /** + * Save + * + * @param oneService oneService info + */ + void save(ApiServiceVo oneService) throws Exception; + + + + void saveByApp(ApiServiceVo apiService) throws Exception; + + /** + * Update + * + * @param oneService oneService info + */ + void update(ApiServiceVo oneService) throws Exception; + + /** + * query + * + * @param apiServiceQuery + * @return + */ + List query(ApiServiceQuery apiServiceQuery) throws ApiServiceQueryException; + + List queryByWorkspaceId(Integer workspaceId, String userName); + + List queryAllTags(String userName,Integer workspaceId); + + + /** + * query + * + * @param scriptPath + * @return + */ + ApiServiceVo queryByScriptPath(String scriptPath); + + Integer queryCountByPath(String scriptPath, String path); + + Integer queryCountByName(String name); + + /** + * enable api + * @param id api record id + * @return + */ + Boolean enableApi(Long id,String userName); + + /** + * disable api + * @param id api record id + * @return + */ + Boolean disableApi(Long id,String userName); + + Boolean deleteApi(Long id,String userName); + + Boolean updateComment(Long id,String comment,String userName); + + ApiServiceVo queryById(Long id,String userName); + + ApiVersionVo getMaxVersion(long serviceId); + + boolean checkUserWorkspace(String userName,Integer workspaceId); + +// List genDataMapApplyContentDatas(ApiServiceVo apiServiceVo, ApiVersionVo apiVersionVo, String metaDtaInfo); + + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/ApiServiceQueryService.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/ApiServiceQueryService.java new file mode 100644 index 000000000..2f98390a9 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/ApiServiceQueryService.java @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.service; + +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceJob; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceToken; +import com.webank.wedatasphere.dss.apiservice.core.bo.LinkisExecuteResult; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceQueryException; +import com.webank.wedatasphere.dss.apiservice.core.vo.*; + +import java.util.List; +import java.util.Map; + + +public interface ApiServiceQueryService { + /** + * 执行查询 + * + * @param path path + * @param params params + * @param moduleName moduleName + * @param httpMethod httpMethod + * @return 查询接口 + */ + LinkisExecuteResult query(String path, Map params, String moduleName, String httpMethod, ApiServiceToken tokenDetail, String loginUser); + + List queryParamList(String scriptPath, Long versionId); + + /** + * 查询api 版本信息 + * + * @param serviceId + * @return + */ + List queryApiVersionById(Long serviceId); + + ApiServiceVo queryByVersionId(String userName,Long versionId) throws ApiServiceQueryException; + + ApiServiceJob getJobByTaskId(String taskId); + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/TokenQueryService.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/TokenQueryService.java new file mode 100644 index 000000000..58e0a3014 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/TokenQueryService.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.service; + +import com.github.pagehelper.PageInfo; +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; +import com.webank.wedatasphere.dss.apiservice.core.bo.TokenQuery; + + +public interface TokenQueryService { + + /** + * 查询 + * */ + PageInfo query(TokenQuery tokenQuery); + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/ApiServiceImpl.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/ApiServiceImpl.java new file mode 100644 index 000000000..3efec44e6 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/ApiServiceImpl.java @@ -0,0 +1,629 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.service.impl; + +import com.google.common.base.Splitter; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceQuery; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceToken; +import com.webank.wedatasphere.dss.apiservice.core.constant.ApiCommonConstant; +import com.webank.wedatasphere.dss.apiservice.core.constant.SQLMetadataInfoCheckStatus; +import com.webank.wedatasphere.dss.apiservice.core.constant.SaveTokenEnum; +import com.webank.wedatasphere.dss.apiservice.core.dao.*; +import com.webank.wedatasphere.dss.apiservice.core.util.UUIDGenerator; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceQueryException; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceTokenException; +import com.webank.wedatasphere.dss.apiservice.core.execute.LinkisJobSubmit; +import com.webank.wedatasphere.dss.apiservice.core.token.JwtManager; +import com.webank.wedatasphere.dss.apiservice.core.token.TokenAuth; +import com.webank.wedatasphere.dss.apiservice.core.vo.*; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiService; +import com.webank.wedatasphere.linkis.bml.client.BmlClient; +import com.webank.wedatasphere.linkis.bml.client.BmlClientFactory; +import com.webank.wedatasphere.linkis.bml.protocol.BmlUpdateResponse; +import com.webank.wedatasphere.linkis.bml.protocol.BmlUploadResponse; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.common.io.FsPath; +import com.webank.wedatasphere.linkis.storage.script.ScriptFsWriter; +import com.webank.wedatasphere.linkis.storage.script.ScriptMetaData; +import com.webank.wedatasphere.linkis.storage.script.ScriptRecord; +import com.webank.wedatasphere.linkis.storage.script.Variable; +import com.webank.wedatasphere.linkis.storage.script.VariableParser; +import com.webank.wedatasphere.linkis.storage.script.writer.StorageScriptFsWriter; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import org.apache.commons.lang.StringUtils; +import org.apache.http.Consts; +import org.apache.ibatis.annotations.Param; +import org.mortbay.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import javax.annotation.PostConstruct; +import java.io.IOException; +import java.io.InputStream; +import java.util.*; +import java.util.stream.Collectors; + + +@Service +public class ApiServiceImpl implements ApiService { + + private static final Logger LOG = LoggerFactory.getLogger(ApiServiceImpl.class); + + + @Autowired + private ApiServiceDao apiServiceDao; + + @Autowired + private ApiServiceParamDao apiServiceParamDao; + + @Autowired + private ApiServiceVersionDao apiServiceVersionDao; + + @Autowired + private ApiServiceTokenManagerDao apiServiceTokenManagerDao; + + @Autowired + TokenAuth tokenAuth; + + + + + /** + * Bml client + */ + private BmlClient client; + + private UJESClient ujesClient; + + @PostConstruct + public void buildClient() { + LOG.info("build client start ======"); + client = BmlClientFactory.createBmlClient(); + Map props = new HashMap<>(); + ujesClient = LinkisJobSubmit.getClient(props); + LOG.info("build client end ======="); + } + + @Override + @Transactional(rollbackFor = Exception.class) + public void save(ApiServiceVo apiService) throws Exception { + String user = apiService.getCreator(); + String resourceId = null; + try { + // check script path if already created + String scriptPath = apiService.getScriptPath(); + + // upload to bml + Map uploadResult = uploadBml(user, scriptPath, + apiService.getMetadata(), apiService.getContent()); + + // insert linkis_oneservice_config + String version = uploadResult.get("version"); + resourceId = uploadResult.get("resourceId"); + apiServiceDao.insert(apiService); + + //insert into version + ApiVersionVo apiVersionVo = new ApiVersionVo(); + apiVersionVo.setApiId(apiService.getId()); + apiVersionVo.setBmlResourceId(resourceId); + apiVersionVo.setBmlVersion(version); + apiVersionVo.setVersion(version); + apiVersionVo.setCreator(user); + apiVersionVo.setCreateTime(Calendar.getInstance().getTime()); + apiVersionVo.setSource(apiService.getScriptPath()); + //1为正常 0为禁用 + apiVersionVo.setStatus(1); + + String approvalNo = UUIDGenerator.genUUID(); + apiVersionVo.setAuthId(approvalNo); + + //顺序不能改变，版本信息依赖审批单信息 + //todo update by query + apiVersionVo.setMetadataInfo("default"); + apiServiceVersionDao.insert(apiVersionVo); + + + + // insert linkis_oneservice_params + List params = apiService.getParams(); + if (params != null && !params.isEmpty()) { + for (ParamVo param : params) { + param.setApiVersionId(apiVersionVo.getId()); + apiServiceParamDao.insert(param); + } + } + + //insert a token record for self + genTokenForPublisher(apiService,apiVersionVo.getId()); + } catch (Exception e) { + LOG.error("one service insert error", e); + if (StringUtils.isNotBlank(resourceId)) { +// removeBml(user, resourceId); + } + if (e.getCause() instanceof ErrorException) { + throw (ErrorException) e.getCause(); + } + throw e; + } + } + + + @Override + @Transactional(rollbackFor = Exception.class) + public void saveByApp(ApiServiceVo apiService) throws Exception { + String user = apiService.getCreator(); + String resourceId = null; + try { + // check script path if already created + String scriptPath = apiService.getScriptPath(); + + // upload to bml + Map uploadResult = uploadBml(user, scriptPath, + apiService.getMetadata(), apiService.getContent()); + + // insert linkis_oneservice_config + String version = uploadResult.get("version"); + resourceId = uploadResult.get("resourceId"); + apiServiceDao.insert(apiService); + + //insert into version + ApiVersionVo apiVersionVo = new ApiVersionVo(); + apiVersionVo.setApiId(apiService.getId()); + apiVersionVo.setBmlResourceId(resourceId); + apiVersionVo.setBmlVersion(version); + apiVersionVo.setVersion(version); + apiVersionVo.setCreator(user); + apiVersionVo.setCreateTime(Calendar.getInstance().getTime()); + apiVersionVo.setSource(apiService.getScriptPath()); + //1为正常 0为禁用 + apiVersionVo.setStatus(1); + + + //生成审批记录，必须使用发布用户执行sql +// checkApprovalFromDM(user,apiService,apiVersionVo); + //顺序不能改变，版本信息依赖审批单信息 + apiServiceVersionDao.insert(apiVersionVo); + +// addApprovalToDB(apiService,apiVersionVo.getId(),apiVersionVo.getAuthId()); + + // insert linkis_oneservice_params + List params = apiService.getParams(); + if (params != null && !params.isEmpty()) { + for (ParamVo param : params) { + param.setApiVersionId(apiVersionVo.getId()); + apiServiceParamDao.insert(param); + } + } + + //insert a token record for self + genTokenForPublisher(apiService,apiVersionVo.getId()); + } catch (Exception e) { + LOG.error("one service insert error", e); + if (StringUtils.isNotBlank(resourceId)) { +// removeBml(user, resourceId); + } + if (e.getCause() instanceof ErrorException) { + throw (ErrorException) e.getCause(); + } + throw e; + } + } + + + + + + @Override + @Transactional(rollbackFor = Exception.class) + public void update(ApiServiceVo apiService) throws Exception { + try { + if(null!=apiService.getTargetServiceId()) { + ApiVersionVo maxTargetApiVersionVo = getMaxVersion(apiService.getTargetServiceId()); + if(!checkUserWorkspace(apiService.getModifier(),apiService.getWorkspaceId().intValue())){ + throw new ApiServiceQueryException(800035,"Only can update the api service by owner workspace! "); + } + if(maxTargetApiVersionVo.getCreator().equals(apiService.getModifier())) { + Map updateResult = updateBml(apiService.getModifier(), maxTargetApiVersionVo.getBmlResourceId(), + apiService.getScriptPath(), apiService.getMetadata(), apiService.getContent()); + apiService.setCreator(maxTargetApiVersionVo.getCreator()); + apiService.setId(maxTargetApiVersionVo.getApiId()); + apiServiceDao.updateToTarget(apiService); + Log.info("Update to other Api Service, ID: " + apiService.getTargetServiceId() + ",resourceId:" + maxTargetApiVersionVo.getBmlResourceId()); + + + String version = updateResult.get("version"); + String resourceId = updateResult.get("resourceId"); + + + //update api version + //insert into version + ApiVersionVo apiServiceVersionVo = new ApiVersionVo(); + apiServiceVersionVo.setApiId(apiService.getId()); + apiServiceVersionVo.setBmlResourceId(resourceId); + apiServiceVersionVo.setBmlVersion(version); + apiServiceVersionVo.setVersion(version); + apiServiceVersionVo.setCreator(apiService.getModifier()); + apiServiceVersionVo.setCreateTime(Calendar.getInstance().getTime()); + apiServiceVersionVo.setSource(apiService.getScriptPath()); + //0默认已禁用 1为正常 + apiServiceVersionVo.setStatus(1); + String approvalNo = UUIDGenerator.genUUID(); + apiServiceVersionVo.setAuthId(approvalNo); + + //todo update by query + apiServiceVersionVo.setMetadataInfo("null"); + //顺序不能改变，版本信息依赖审批单信息 + apiServiceVersionDao.insert(apiServiceVersionVo); + + //改变历史版本状态 + apiServiceVersionDao.updateApiVersionStatusById(maxTargetApiVersionVo.getId(), 0); + + //改变历史Token状态 + apiServiceTokenManagerDao.disableTokenStatusByVersionId(maxTargetApiVersionVo.getId()); + // insert params + List params = apiService.getParams(); + if (params != null && !params.isEmpty()) { + for (ParamVo param : params) { + param.setApiVersionId(apiServiceVersionVo.getId()); + apiServiceParamDao.insert(param); + } + } + +// addApprovalToDB(apiService, apiServiceVersionVo.getId(), apiServiceVersionVo.getAuthId()); + + //insert a token record for self + genTokenForPublisher(apiService, apiServiceVersionVo.getId()); + }else { + throw new ApiServiceQueryException(800036,"Only can update the api service by owner! "); + } + }else { + throw new ApiServiceQueryException(800037,"Target service id can not be null for update"); + } + + } catch (Exception e) { + LOG.error("api service update error", e); + if (e.getCause() instanceof ErrorException) { + throw (ErrorException) e.getCause(); + } + throw e; + } + } + + @Override + public List query(ApiServiceQuery apiServiceQuery) throws ApiServiceQueryException { + //todo 查询需要优化，量小时不影响效率 + List queryList = apiServiceDao.query(apiServiceQuery); + + //add auth check + List userTokenManagerVos = apiServiceTokenManagerDao.queryByApplyUser(apiServiceQuery.getUserName()); + + List authQueryList = queryList.stream().filter(apiServiceVo -> { + TokenManagerVo findUserTokenManagerVo = userTokenManagerVos.stream().filter(userTokenManagerVo -> + userTokenManagerVo.getApiId().equals(apiServiceVo.getId()) + && userTokenManagerVo.getUser().equals(apiServiceQuery.getUserName()) + ).findAny().orElse(null); + if (null != findUserTokenManagerVo) { + //检查token状态为有效的 + if(findUserTokenManagerVo.getStatus().equals(1) || findUserTokenManagerVo.getPublisher().equals(apiServiceVo.getCreator())) { + return true; + }else { + return false; + } + } else { + return false; + } + }).collect(Collectors.toList()); + + // query param + if (authQueryList != null && !authQueryList.isEmpty()) { + for (ApiServiceVo apiServiceVo : authQueryList) { + ApiVersionVo maxApiVersionVo = getMaxVersion(apiServiceVo.getId()); + if(null == maxApiVersionVo){ + throw new ApiServiceQueryException(800032,"数据服务API版本记录为空"+apiServiceVo.getName()); + } + apiServiceVo.setParams(apiServiceParamDao.queryByVersionId(maxApiVersionVo.getId())); + apiServiceVo.setLatestVersionId(maxApiVersionVo.getId()); + } + } + + return authQueryList; + } + + @Override + public List queryByWorkspaceId(Integer workspaceId, String userName){ + List result = apiServiceDao.queryByWorkspaceId(workspaceId,userName); + return result; + } + + + @Override + public ApiServiceVo queryById(Long id,String userName) { + ApiServiceVo apiServiceVo = apiServiceDao.queryById(id); + ApiVersionVo maxApiVersionVo = getMaxVersion(apiServiceVo.getId()); + List userTokenManagerVos = apiServiceTokenManagerDao.queryByApplyUserAndVersionId(userName,maxApiVersionVo.getId()); + if(userTokenManagerVos.size()>0) { + // query param + if (apiServiceVo != null) { + apiServiceVo.setParams(apiServiceParamDao.queryByVersionId(maxApiVersionVo.getId())); + apiServiceVo.setUserToken(userTokenManagerVos.get(0).getToken()); + apiServiceVo.setLatestVersionId(maxApiVersionVo.getId()); + } + }else { + return null; + } + return apiServiceVo; + } + + + @Override + public List queryAllTags(String userName,Integer workspaceId) { + //todo 会有历史版本的tag + List tags = apiServiceDao.queryAllTags(userName,workspaceId); + List tagList= + tags.stream().filter(tag->!StringUtils.isEmpty(tag)).map(tag -> Splitter.on(",").splitToList(tag)).flatMap(List::stream).distinct() + .collect(Collectors.toList()); + return tagList; + } + + + @Override + public ApiServiceVo queryByScriptPath(@Param("scriptPath") String scriptPath) { + List apiServiceList = apiServiceDao.queryByScriptPath(scriptPath); + ApiServiceVo latestApiService = apiServiceList.stream().max(Comparator.comparing(ApiServiceVo::getModifyTime)).orElse(null); + if(null == latestApiService){ + return null; + } + List apiVersionVos =apiServiceVersionDao.queryApiVersionByApiServiceId(latestApiService.getId()); + ApiVersionVo maxVersion =apiVersionVos.stream().max(Comparator.comparing(ApiVersionVo::getVersion)).orElse(null); + // query param + if (latestApiService != null && null != maxVersion ) { + latestApiService.setParams(apiServiceParamDao.queryByVersionId(maxVersion.getId())); + latestApiService.setLatestVersionId(maxVersion.getId()); + } + return latestApiService; + } + + + + + + + @Override + public Integer queryCountByPath(String scriptPath, String path) { + return apiServiceDao.queryCountByPath(scriptPath, path); + } + + @Override + public Integer queryCountByName(String name) { + return apiServiceDao.queryCountByName(name); + } + + @Override + public Boolean enableApi(Long id,String userName) { + ApiServiceVo apiServiceVo = apiServiceDao.queryById(id); + if(!checkUserWorkspace(userName,apiServiceVo.getWorkspaceId().intValue())){ + LOG.error("api service check workspace error"); + return false; + } + if(apiServiceVo.getCreator().equals(userName)) { + + Integer updateCount = apiServiceDao.enableApi(id); + List targetApiVersionList = apiServiceVersionDao.queryApiVersionByApiServiceId(id); + ApiVersionVo maxTargetApiVersionVo = targetApiVersionList.stream().max(Comparator.comparing(ApiVersionVo::getVersion)).orElse(null); + + apiServiceTokenManagerDao.enableTokenStatusByVersionId(maxTargetApiVersionVo.getId()); + apiServiceVersionDao.updateApiVersionStatusById(maxTargetApiVersionVo.getId(), 1); + return updateCount > 0; + }else { + return false; + } + } + + @Override + public Boolean disableApi(Long id,String userName) { + ApiServiceVo apiServiceVo = apiServiceDao.queryById(id); + if(!checkUserWorkspace(userName,apiServiceVo.getWorkspaceId().intValue())){ + LOG.error("api service check workspace error"); + return false; + } + if(apiServiceVo.getCreator().equals(userName)) { + Integer updateCount = apiServiceDao.disableApi(id); + apiServiceTokenManagerDao.disableTokenStatusByApiId(id); + apiServiceVersionDao.updateAllApiVersionStatusByApiServiceId(id, 0); + return updateCount > 0; + }else { + return false; + } + } + + + @Override + public Boolean deleteApi(Long id,String userName) { + ApiServiceVo apiServiceVo = apiServiceDao.queryById(id); + if(!checkUserWorkspace(userName,apiServiceVo.getWorkspaceId().intValue())){ + LOG.error("api service check workspace error"); + return false; + } + if(apiServiceVo.getCreator().equals(userName)) { + Integer updateCount = apiServiceDao.deleteApi(id); + apiServiceTokenManagerDao.disableTokenStatusByApiId(id); + apiServiceVersionDao.updateAllApiVersionStatusByApiServiceId(id, 0); + return updateCount > 0; + }else { + return false; + } + } + + @Override + public Boolean updateComment(Long id, String comment, String userName) { + ApiServiceVo apiServiceVo = apiServiceDao.queryById(id); + if(!checkUserWorkspace(userName,apiServiceVo.getWorkspaceId().intValue())){ + LOG.error("api service check workspace error"); + return false; + } + if(apiServiceVo.getCreator().equals(userName)) { + Integer updateCount = apiServiceDao.updateApiServiceComment(id,comment); + return updateCount > 0; + }else { + return false; + } + } + + private Map uploadBml(String userName, String scriptPath, Map metadata, String scriptContent) { + try { + ScriptFsWriter writer = StorageScriptFsWriter.getScriptFsWriter(new FsPath(scriptPath), Consts.UTF_8.toString(), null); + List variableList=null; + if(metadata.entrySet().size() >0) { + Variable[] v = VariableParser.getVariables(metadata); + variableList = Arrays.stream(v).filter(var -> !StringUtils.isEmpty(var.value())).collect(Collectors.toList()); + + } + if(variableList!=null) { + writer.addMetaData(new ScriptMetaData(variableList.toArray(new Variable[0]))); + }else { + writer.addMetaData(null); + } + writer.addRecord(new ScriptRecord(scriptContent)); + InputStream inputStream = writer.getInputStream(); + // 新增文件 + BmlUploadResponse resource = client.uploadResource(userName, scriptPath, inputStream); + if (!resource.isSuccess()) { + throw new IOException("upload bml error"); + } + Map result = new HashMap<>(); + result.put("resourceId", resource.resourceId()); + result.put("version", resource.version()); + return result; + } catch (IOException e) { + LOG.error("upload bml error", e); + throw new RuntimeException(e); + } + } + + private Map updateBml(String userName, String resourceId, String scriptPath, Map metadata, String scriptContent) { + try { + ScriptFsWriter writer = StorageScriptFsWriter.getScriptFsWriter(new FsPath(scriptPath), Consts.UTF_8.toString(), null); + Variable[] v = VariableParser.getVariables(metadata); + List variableList = Arrays.stream(v).filter(var -> !StringUtils.isEmpty(var.value())).collect(Collectors.toList()); + writer.addMetaData(new ScriptMetaData(variableList.toArray(new Variable[0]))); + writer.addRecord(new ScriptRecord(scriptContent)); + InputStream inputStream = writer.getInputStream(); + + // 更新文件 + BmlUpdateResponse resource = client.updateResource(userName, resourceId, "", inputStream); + if (!resource.isSuccess()) { + throw new IOException("update bml error"); + } + Map result = new HashMap<>(); + result.put("resourceId", resource.resourceId()); + result.put("version", resource.version()); + return result; + } catch (IOException e) { + LOG.error("update bml error", e); + throw new RuntimeException(e); + } + } + + private void removeBml(String userName, String resourceId) { + try { + LOG.info("delete bml resource: userName: " + userName + ", resourceId: " + resourceId); + client.deleteResource(userName, resourceId); + } catch (Exception e) { + LOG.error("remove bml error", e); + } + } + + + public SQLMetadataInfoCheckStatus sendApprovalToDM(String user, ApiServiceVo apiService, ApiVersionVo apiVersionVo) throws Exception { + + String approvalNo = UUIDGenerator.genUUID(); + apiVersionVo.setAuthId(approvalNo); + return SQLMetadataInfoCheckStatus.LEGAL; + } + + + + + public void genTokenForPublisher(ApiServiceVo apiService,Long apiVersionId) throws ApiServiceTokenException { + + List tokenManagerVoList = genTokenForAccessApi(apiService,apiVersionId); + SaveTokenEnum res = tokenAuth.saveTokensToDb(tokenManagerVoList, ApiCommonConstant.DEFAULT_APPROVAL_NO); + } + + /** + * 为申请用户和自己生成访问token + * @param apiService + * @param apiVersionId + * @return + */ + private List genTokenForAccessApi(ApiServiceVo apiService,Long apiVersionId){ + List tokenManagerVoList = new ArrayList<>(); + List userNamesTmp = Arrays.asList(apiService.getApprovalVo().getApplyUser().split(",")); + List userNames = new ArrayList<>(userNamesTmp); + userNames.add(apiService.getCreator()); + userNames.stream().forEach(userName ->{ + TokenManagerVo tokenManagerVo = new TokenManagerVo(); + tokenManagerVo.setApiId(apiService.getId()); + tokenManagerVo.setApiVersionId(apiVersionId); + tokenManagerVo.setApplyTime(Calendar.getInstance().getTime()); + tokenManagerVo.setDuration(365L); + tokenManagerVo.setReason("create api service"); + tokenManagerVo.setStatus(1); + tokenManagerVo.setIpWhitelist(""); + tokenManagerVo.setCaller("scriptis"); + tokenManagerVo.setUser(userName); + tokenManagerVo.setPublisher(apiService.getCreator()); + + ApiServiceToken apiServiceToken = new ApiServiceToken(); + apiServiceToken.setApplyUser(userName); + apiServiceToken.setPublisher(apiService.getCreator()); + apiServiceToken.setApplyTime(tokenManagerVo.getApplyTime()); + apiServiceToken.setApiServiceId(tokenManagerVo.getApiId()); + + tokenManagerVo.setToken(JwtManager.createToken(userName,apiServiceToken,tokenManagerVo.getDuration())); + + tokenManagerVo.setApplySource(ApiCommonConstant.DEFAULT_APPROVAL_NO); + tokenManagerVoList.add(tokenManagerVo); + }); + + return tokenManagerVoList; + } + + + @Override + public ApiVersionVo getMaxVersion(long serviceId){ + List apiVersionVoList = apiServiceVersionDao.queryApiVersionByApiServiceId(serviceId); + ApiVersionVo maxApiVersionVo = apiVersionVoList.stream().max(Comparator.comparing(ApiVersionVo::getVersion)).orElse(null); + return maxApiVersionVo; + } + + @Override + public boolean checkUserWorkspace(String userName,Integer workspaceId){ + //todo cache user workspaceIds + return true; +// String workspaceIds = ExecuteCodeHelper.getUserWorkspaceIds(userName,ujesClient); +// if(Arrays.stream(workspaceIds.split(",")).map(Integer::valueOf).collect(Collectors.toList()).contains(workspaceId)){ +// return true; +// }else { +// return false; +// } + + } + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/ApiServiceQueryServiceImpl.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/ApiServiceQueryServiceImpl.java new file mode 100644 index 000000000..cccdeef52 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/ApiServiceQueryServiceImpl.java @@ -0,0 +1,546 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.service.impl; + +import com.google.common.cache.Cache; +import com.google.common.cache.CacheBuilder; +import com.google.common.cache.RemovalCause; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceJob; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceToken; +import com.webank.wedatasphere.dss.apiservice.core.bo.LinkisExecuteResult; +import com.webank.wedatasphere.dss.apiservice.core.config.ApiServiceConfiguration; +import com.webank.wedatasphere.dss.apiservice.core.constant.ParamType; +import com.webank.wedatasphere.dss.apiservice.core.constant.ParamTypeEnum; +import com.webank.wedatasphere.dss.apiservice.core.constant.RequireEnum; +import com.webank.wedatasphere.dss.apiservice.core.dao.*; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiExecuteException; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceQueryException; +import com.webank.wedatasphere.dss.apiservice.core.execute.ApiServiceExecuteJob; +import com.webank.wedatasphere.dss.apiservice.core.execute.DefaultApiServiceJob; +import com.webank.wedatasphere.dss.apiservice.core.execute.ExecuteCodeHelper; +import com.webank.wedatasphere.dss.apiservice.core.execute.LinkisJobSubmit; +import com.webank.wedatasphere.dss.apiservice.core.jdbc.DatasourceService; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiService; +import com.webank.wedatasphere.dss.apiservice.core.util.DateUtil; +import com.webank.wedatasphere.dss.apiservice.core.util.SQLCheckUtil; +import com.webank.wedatasphere.dss.apiservice.core.vo.*; +//import com.webank.wedatasphere.dss.oneservice.core.jdbc.JdbcUtil; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceRuntimeException; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiServiceQueryService; +import com.webank.wedatasphere.dss.apiservice.core.util.AssertUtil; +import com.webank.wedatasphere.dss.apiservice.core.util.ModelMapperUtil; +//import com.webank.wedatasphere.dss.oneservice.core.vo.*; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiServiceVo; +import com.webank.wedatasphere.linkis.bml.client.BmlClient; +import com.webank.wedatasphere.linkis.bml.client.BmlClientFactory; +import com.webank.wedatasphere.linkis.bml.protocol.BmlDownloadResponse; +import com.webank.wedatasphere.linkis.common.io.FsPath; +import com.webank.wedatasphere.linkis.storage.source.FileSource; +import com.webank.wedatasphere.linkis.storage.source.FileSource$; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; +import org.apache.commons.collections4.MapUtils; +import org.apache.commons.lang.StringUtils; +import org.apache.commons.math3.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.jdbc.core.RowMapper; +import org.springframework.jdbc.support.JdbcUtils; +import org.springframework.stereotype.Service; +import org.springframework.util.LinkedCaseInsensitiveMap; +import scala.Tuple3; + +import javax.annotation.PostConstruct; +import java.io.IOException; +import java.io.InputStream; +import java.sql.ResultSet; +import java.sql.ResultSetMetaData; +import java.sql.SQLException; +import java.util.*; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; + +import static java.util.stream.Collectors.toMap; + + +@Service +public class ApiServiceQueryServiceImpl implements ApiServiceQueryService { + private static final Logger LOG = LoggerFactory.getLogger(ApiServiceQueryServiceImpl.class); + + + Map runJobs = new HashMap<>(); + + /** + * key:resourceId+version + * value:bml + */ + private static Cache>> bmlCache = CacheBuilder.newBuilder() + .expireAfterWrite(6, TimeUnit.HOURS) + .maximumSize(10000) + .removalListener((notification) -> { + if (notification.getCause() == RemovalCause.SIZE) { + LOG.warn("bml缓存容量不足，移除key:" + notification.getKey()); + } + }) + .build(); + + /** + * key:resourceId+version + * value:configParam + */ + private static Cache> configParamCache = CacheBuilder.newBuilder() + .expireAfterWrite(6, TimeUnit.HOURS) + .maximumSize(10000) + .removalListener((notification) -> { + if (notification.getCause() == RemovalCause.SIZE) { + LOG.warn("configParamCache缓存容量不足，移除key:" + notification.getKey()); + } + }) + .build(); + + /** + * key:datasourceMap + * value:jdbc连接信息 + */ + private static Cache, Tuple3> datasourceCache = CacheBuilder.newBuilder() + .expireAfterWrite(1, TimeUnit.MINUTES) + .maximumSize(2000) + .removalListener((notification) -> { + if (notification.getCause() == RemovalCause.SIZE) { + LOG.warn("datasource缓存容量不足，移除key:" + notification.getKey()); + } + }) + .build(); + + @Autowired + private ApiServiceDao apiServiceDao; + + @Autowired + private ApiServiceParamDao apiServiceParamDao; + + @Autowired + private DatasourceService datasourceService; + + @Autowired + private ApiServiceVersionDao apiServiceVersionDao; + + @Autowired + private ApiServiceTokenManagerDao apiServiceTokenManagerDao; + + + @Autowired + private ApiService apiService; + + + @Autowired + private ApiServiceAccessDao apiServiceAccessDao; + + /** + * Bml client + */ + private BmlClient client; + + @PostConstruct + public void init() { + LOG.info("build client start ======"); + client = BmlClientFactory.createBmlClient(); + LOG.info("build client end ======="); + } + + @Override + public LinkisExecuteResult query(String path, + Map reqParams, + String moduleName, + String httpMethod, + ApiServiceToken tokenDetail, + String loginUser) { + // 根据path查询resourceId和version + + // 得到metadata + + // 执行查询 + //path 必须唯一 + ApiServiceVo apiServiceVo = apiServiceDao.queryByPath(path); + if(null == apiServiceVo){ + throw new ApiServiceRuntimeException("根据脚本路径未匹配到数据服务！"); + } + if(!apiService.checkUserWorkspace(loginUser,apiServiceVo.getWorkspaceId().intValue())){ + throw new ApiServiceRuntimeException("用户工作空间检查不通过！"); + } + if(!apiServiceVo.getId().equals(tokenDetail.getApiServiceId())){ + throw new ApiServiceRuntimeException("用户token中服务ID不匹配！"); + } + + ApiVersionVo maxApiVersionVo =apiService.getMaxVersion(apiServiceVo.getId()); + + AssertUtil.notNull(apiServiceVo, "接口不存在，path=" + path); + AssertUtil.isTrue(StringUtils.equals(httpMethod, apiServiceVo.getMethod().toUpperCase()), + "该接口不支持" + httpMethod + "请求，请用" + apiServiceVo.getMethod() + "请求"); + AssertUtil.isTrue(1 == apiServiceVo.getStatus(), "接口已禁用"); + AssertUtil.notNull(maxApiVersionVo, "未找到最新的版本，path=" + path); + + try { + Pair> collect = queryBml(apiServiceVo.getCreator(), maxApiVersionVo.getBmlResourceId(), + maxApiVersionVo.getBmlVersion(), apiServiceVo.getScriptPath()); + String executeCode = collect.getSecond().get(0)[0]; + + Map variable = (Map) ((Map) collect.getFirst()).get("variable"); + + //没有传入的参数，使用默认值 + Map paramTypes = queryConfigParam(apiServiceVo.getId(), maxApiVersionVo.getVersion()); + if (variable != null) { + variable.forEach((k, v) -> { + if (!reqParams.containsKey(k)) { + if (ParamType.number.equals(paramTypes.get(k))) { + reqParams.put(k, Integer.valueOf(v.toString())); + } else { + reqParams.put(k, v); + } + } + }); + } + + // 用户请求的参数值注入检查，排除token + for(String k: reqParams.keySet()){ + if(!k.equals(ApiServiceConfiguration.API_SERVICE_TOKEN_KEY.getValue()) + && SQLCheckUtil.doParamInjectionCheck(reqParams.get(k).toString())) { + // 如果注入直接返回null + LOG.warn("用户参数存在非法的关键字{}", reqParams.get(k).toString()); + return null; + } + } + + //数组类型，如果没有加单引号，自动添加 + reqParams.forEach((k,v) ->{ + if(ParamType.array.equals(paramTypes.get(k))){ + String sourceStr = v.toString(); + String targetStr =sourceStr; + sourceStr= sourceStr.replaceAll("(\n\r|\r\n|\r|\n)", ","); + sourceStr= sourceStr.replaceAll(",,", ","); + + if(!sourceStr.contains("\'")){ + targetStr= Arrays.stream(sourceStr.split(",")).map(s -> "\'" + s + "\'").collect(Collectors.joining(",")); + reqParams.put(k, targetStr); + }else { + reqParams.put(k, sourceStr); + } + } + }); + + +// AssertUtil.isTrue(MapUtils.isNotEmpty((Map) collect.getKey()), "数据源不能为空"); + + + ApiServiceExecuteJob job = new DefaultApiServiceJob(); + //sql代码封装成scala执行 + job.setCode(ExecuteCodeHelper.packageCodeToExecute(executeCode, maxApiVersionVo.getMetadataInfo())); + job.setEngineType(apiServiceVo.getType()); + job.setRunType("scala"); + //不允许创建用户自己随意代理执行，创建用户只能用自己用户执行 + //如果需要代理执行可以在这里更改用户 + job.setUser(loginUser); + + job.setParams(null); + job.setRuntimeParams(reqParams); + job.setScriptePath(apiServiceVo.getScriptPath()); + UJESClient ujesClient = LinkisJobSubmit.getClient(paramTypes); + + //记录用户Api被执行信息 + ApiAccessVo apiAccessVo = new ApiAccessVo(); + apiAccessVo.setUser(loginUser); + apiAccessVo.setApiPublisher(apiServiceVo.getCreator()); + apiAccessVo.setApiServiceName(apiServiceVo.getName()); + apiAccessVo.setApiServiceId(apiServiceVo.getId()); + apiAccessVo.setApiServiceVersionId(maxApiVersionVo.getId()); + apiAccessVo.setProxyUser(job.getUser()); + apiAccessVo.setAccessTime(DateUtil.getNow()); + apiServiceAccessDao.addAccessRecord(apiAccessVo); + + + JobExecuteResult jobExecuteResult = LinkisJobSubmit.execute(job,ujesClient); + + //记录执行任务用户和代理用户关系，没有代理用户的统一设置为登录用户 + ApiServiceJob apiServiceJob = new ApiServiceJob(); + apiServiceJob.setSubmitUser(loginUser); + apiServiceJob.setProxyUser(job.getUser()); + apiServiceJob.setJobExecuteResult(jobExecuteResult); + runJobs.put(jobExecuteResult.getTaskID(),apiServiceJob); + + + LinkisExecuteResult linkisExecuteResult = new LinkisExecuteResult(jobExecuteResult.getTaskID(), jobExecuteResult.getExecID()); + return linkisExecuteResult; + } catch (IOException e) { + throw new ApiServiceRuntimeException(e.getMessage(), e); + } + } + + + + + @Override + public ApiServiceVo queryByVersionId(String userName,Long versionId) throws ApiServiceQueryException { + ApiVersionVo apiVersionVo = apiServiceVersionDao.queryApiVersionByVersionId(versionId); + ApiServiceVo apiServiceVo = apiServiceDao.queryById(apiVersionVo.getApiId()); + //授权后才可以查看内容 + List userTokenManagerVos = apiServiceTokenManagerDao.queryByApplyUserAndVersionId(userName,versionId); + if(userTokenManagerVos.size()>0) { + try { + Pair> collect = queryBml(apiServiceVo.getCreator(), apiVersionVo.getBmlResourceId(), + apiVersionVo.getBmlVersion(), apiServiceVo.getScriptPath()); + String executeCode = collect.getSecond().get(0)[0]; + apiServiceVo.setContent(executeCode); + + } catch (IOException e) { + throw new ApiServiceQueryException(800002, "查询数据服务API内容异常"); + } + apiServiceVo.setScriptPath(apiVersionVo.getSource()); + return apiServiceVo; + }else { + + throw new ApiServiceQueryException(800003, "没有权限查看数据服务API内容，请先提单授权"); + } + } + + @Override + public List queryParamList(String scriptPath, Long versionId) { + ApiVersionVo targetApiVersionVo = apiServiceVersionDao.queryApiVersionByVersionId(versionId); + + ApiServiceVo apiServiceVo=apiServiceDao.queryById(targetApiVersionVo.getApiId()); + + AssertUtil.notNull(apiServiceVo, "接口不存在，path=" + scriptPath); + + AssertUtil.notNull(targetApiVersionVo, "目标参数版本不存在，path=" + scriptPath+",version:"+versionId); + + // todo~！ + List paramVoList = apiServiceParamDao.queryByVersionId(targetApiVersionVo.getId()); + + + List queryParamVoList = new ArrayList<>(); + + Map paramMap = paramVoList.stream() + .collect(Collectors.toMap(ParamVo::getName, k -> k, (k, v) -> k)); + Map variableMap = getVariable(apiServiceVo,versionId); + paramMap.keySet() + .forEach(keyItem -> { + ParamVo paramVo = paramMap.get(keyItem); + QueryParamVo queryParamVo = ModelMapperUtil.strictMap(paramVo, QueryParamVo.class); + queryParamVo.setTestValue(variableMap.containsKey(keyItem) ? variableMap.get(keyItem).toString() : ""); + queryParamVo.setRequireStr(RequireEnum.getEnum(paramVo.getRequired()).getName()); + queryParamVo.setType(paramVo.getType()); + + queryParamVoList.add(queryParamVo); + }); + + return queryParamVoList; + } + + @Override + public List queryApiVersionById(Long serviceId) { + List apiVersionVoList = apiServiceVersionDao.queryApiVersionByApiServiceId(serviceId); + return apiVersionVoList; + } + + private Map getVariable(ApiServiceVo apiServiceVo,Long versionId) { + Map variableMap = null; + ApiVersionVo apiVersionVo = apiServiceVersionDao.queryApiVersionByVersionId(versionId); + if(null != apiServiceVo) { + try { + Pair> collect = queryBml(apiServiceVo.getCreator(), apiVersionVo.getBmlResourceId(), + apiVersionVo.getBmlVersion(), apiServiceVo.getScriptPath()); + + variableMap = (Map) ((Map) collect.getFirst()).get("variable"); + } catch (IOException e) { + throw new ApiServiceRuntimeException(e.getMessage(), e); + } + } + return null == variableMap ? Collections.EMPTY_MAP : variableMap; + } + + private Pair> queryBml(String userName, String resourceId, String version, + String scriptPath) throws IOException { + String key = String.join("-", resourceId, version); + Pair> collect = bmlCache.getIfPresent(key); + + if (collect == null) { + synchronized (this) { + collect = bmlCache.getIfPresent(key); + if (collect == null) { + BmlDownloadResponse resource; + if (version == null) { + resource = client.downloadResource(userName, resourceId, null); + } else { + resource = client.downloadResource(userName, resourceId, version); + } + + AssertUtil.isTrue(resource.isSuccess(), "查询bml错误"); + + InputStream inputStream = resource.inputStream(); + + try (FileSource fileSource = FileSource$.MODULE$.create(new FsPath(scriptPath), inputStream)) { + //todo 数组取了第一个 + collect = fileSource.collect()[0]; + bmlCache.put(key, collect); + } + } + } + } + + + return collect; + } + + private Map queryConfigParam(long apiId, String version) { + String key = String.join("-", apiId + "", version); + Map collect = configParamCache.getIfPresent(key); + + if (collect == null) { + synchronized (this) { + collect = configParamCache.getIfPresent(key); + if (collect == null) { + List apiVersionVoList = apiServiceVersionDao.queryApiVersionByApiServiceId(apiId); + ApiVersionVo apiVersionVo = apiVersionVoList.stream().filter(apiVersionVoTmp -> apiVersionVoTmp.getVersion().equals(version)).findFirst().orElse(null); + + collect = apiServiceParamDao.queryByVersionId(apiVersionVo.getId()) + .stream() + .collect(toMap(ParamVo::getName, ParamVo::getType)); + configParamCache.put(key, collect); + } + } + } + + return collect; + } + + + + +// private Tuple3 getDatasourceInfo(final Map datasourceMap) { +// Tuple3 tuple3 = datasourceCache.getIfPresent(datasourceMap); +// +// if (tuple3 == null) { +// synchronized (this) { +// tuple3 = datasourceCache.getIfPresent(datasourceMap); +// if (tuple3 == null) { +// tuple3 = JdbcUtil.getDatasourceInfo(datasourceMap); +// datasourceCache.put(datasourceMap, tuple3); +// } +// } +// } +// +// return tuple3; +// } + +// private List> executeJob(String executeCode, +// Object datasourceMap, Map params) { +// +//// Tuple3 tuple3 = getDatasourceInfo((Map) datasourceMap); +//// final String jdbcUrl = tuple3._1().toString(); +//// final String username = tuple3._2().toString(); +//// final String password = tuple3._3().toString(); +// +//// NamedParameterJdbcTemplate namedParameterJdbcTemplate = datasourceService.getNamedParameterJdbcTemplate(jdbcUrl, username, password); +// +// String namedSql = genNamedSql(executeCode, params); +// +//// return namedParameterJdbcTemplate.query(namedSql, new MapSqlParameterSource(params), new ColumnAliasMapRowMapper()); +// +// } + + private static String genNamedSql(String executeCode, Map params) { + // 没有参数，无需生成namedSql + if (MapUtils.isEmpty(params)) { + return executeCode; + } + + for (String paramName : params.keySet()) { + for (String $name : new String[]{"'${" + paramName + "}'", "${" + paramName + "}", "\"${" + paramName + "}\""}) { + if (executeCode.contains($name)) { + executeCode = StringUtils.replace(executeCode, $name, ":" + paramName); + break; + } + } + } + + return executeCode; + } + + + public static class ColumnAliasMapRowMapper implements RowMapper> { + @Override + public Map mapRow(ResultSet rs, int rowNum) throws SQLException { + ResultSetMetaData rsmd = rs.getMetaData(); + int columnCount = rsmd.getColumnCount(); + Map mapOfColValues = createColumnMap(columnCount); + Map mapOfColSuffix = new LinkedCaseInsensitiveMap<>(columnCount); + for (int i = 1; i <= columnCount; i++) { + String key = getColumnKey(JdbcUtils.lookupColumnName(rsmd, i)); + if (mapOfColValues.containsKey(key)) { + if (!mapOfColSuffix.containsKey(key)) { + mapOfColSuffix.put(key, 1); + } else { + mapOfColSuffix.put(key, mapOfColSuffix.get(key) + 1); + } + + key = key + "_" + mapOfColSuffix.get(key); + } + + Object obj = getColumnValue(rs, i); + mapOfColValues.put(key, obj); + } + return mapOfColValues; + } + + protected Map createColumnMap(int columnCount) { + return new LinkedCaseInsensitiveMap<>(columnCount); + } + + protected String getColumnKey(String columnName) { + return columnName; + } + + protected Object getColumnValue(ResultSet rs, int index) throws SQLException { + return JdbcUtils.getResultSetValue(rs, index); + } + + } + + @Override + public ApiServiceJob getJobByTaskId(String taskId){ + ApiServiceJob apiServiceJob=runJobs.get(taskId); + return apiServiceJob; + } + + + private static String getRunTypeFromScriptsPath(String scriptsPath) { + + String res = "sql"; + String fileFlag = scriptsPath.substring(scriptsPath.lastIndexOf(".") + 1); + switch (fileFlag) { + case "sh": + res = "shell"; + break; + case "py": + res= "pyspark"; + break; + default: + res = fileFlag; + break; + } + return res; + + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/TokenQueryServiceImpl.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/TokenQueryServiceImpl.java new file mode 100644 index 000000000..f5dc98e93 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/service/impl/TokenQueryServiceImpl.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.service.impl; + +import com.github.pagehelper.PageHelper; +import com.github.pagehelper.PageInfo; +import com.webank.wedatasphere.dss.apiservice.core.dao.ApiServiceTokenManagerDao; +import com.webank.wedatasphere.dss.apiservice.core.service.TokenQueryService; +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; +import com.webank.wedatasphere.dss.apiservice.core.bo.TokenQuery; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import java.util.List; + +@Service +public class TokenQueryServiceImpl implements TokenQueryService { + private static final Logger LOG = LoggerFactory.getLogger(TokenQueryServiceImpl.class); + + @Autowired + ApiServiceTokenManagerDao apiServiceTokenManagerDao; + + @Override + public PageInfo query(TokenQuery tokenQuery) { + PageHelper.startPage(tokenQuery.getCurrentPage(), tokenQuery.getPageSize()); + List tokenList = apiServiceTokenManagerDao.query(tokenQuery); + LOG.info("token查询的结果列表大小为{}", tokenList.size()); + PageInfo pageInfo = new PageInfo<>(tokenList); + return pageInfo; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/DataMapTokenImpl.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/DataMapTokenImpl.java new file mode 100644 index 000000000..bb32448d8 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/DataMapTokenImpl.java @@ -0,0 +1,109 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.token; + +import com.webank.wedatasphere.dss.apiservice.core.constant.ApiCommonConstant; +import com.webank.wedatasphere.dss.apiservice.core.constant.SaveTokenEnum; +import com.webank.wedatasphere.dss.apiservice.core.dao.ApiServiceTokenManagerDao; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceTokenException; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceToken; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApprovalVo; +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import java.util.*; + + + +@Component +public class DataMapTokenImpl implements TokenAuth { + + private static final Logger logger = LoggerFactory.getLogger(DataMapTokenImpl.class); + + + @Autowired + ApiServiceTokenManagerDao apiServiceTokenManagerDao; + + @Override + public SaveTokenEnum saveTokensToDb(List tokenManagerVos, String approvalNo) throws ApiServiceTokenException { + boolean isEmptyToken = tokenManagerVos.stream().filter(tokenManagerVo -> StringUtils.isEmpty(tokenManagerVo.getToken())).count()>0; + if(checkDuplicateAuth(approvalNo) && approvalNo != ApiCommonConstant.DEFAULT_APPROVAL_NO){ +// throw new ApiServiceTokenException(800001,"ApprovalNo has been repeatedly authorized "); + return SaveTokenEnum.DUPLICATE; + }else if(isEmptyToken){ + throw new ApiServiceTokenException(800001,"Failed to save to db for Some token is empty"); + } + else { + + try { + apiServiceTokenManagerDao.insertList(tokenManagerVos); + } catch (Exception e) { + logger.error("Batch save token to db failed", e); + return SaveTokenEnum.FAILED; + } + } + + return SaveTokenEnum.SUCCESS; + } + + private boolean checkDuplicateAuth(String approvalNo){ + if(apiServiceTokenManagerDao.queryApprovalNo(approvalNo) > 0){ + return true; + }else { + return false; + } + } + + @Override + public List genTokenRecord(ApprovalVo approvalVo){ + + List tokenManagerVoList = new ArrayList<>(); + Arrays.stream(approvalVo.getApplyUser().split(",")).forEach(tempUser ->{ + + TokenManagerVo tmpToken = new TokenManagerVo(); + tmpToken.setApiId(approvalVo.getApiId()); + tmpToken.setApplyTime(new Date()); + tmpToken.setDuration(365L); + tmpToken.setReason("approval token auth"); + tmpToken.setStatus(1); + tmpToken.setIpWhitelist(""); + tmpToken.setCaller("scripts"); + tmpToken.setUser(tempUser); + tmpToken.setPublisher(approvalVo.getCreator()); + tmpToken.setApiVersionId(approvalVo.getApiVersionId()); + + ApiServiceToken apiServiceToken = new ApiServiceToken(); + apiServiceToken.setApplyUser(tempUser); + apiServiceToken.setPublisher(approvalVo.getCreator()); //todo creator + apiServiceToken.setApplyTime(tmpToken.getApplyTime()); + apiServiceToken.setApiServiceId(approvalVo.getApiId()); + + tmpToken.setToken(JwtManager.createToken(tempUser,apiServiceToken,tmpToken.getDuration())); + //审批单号 + tmpToken.setApplySource(approvalVo.getApprovalNo()); + tokenManagerVoList.add(tmpToken); + + + }); + + return tokenManagerVoList; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/JwtManager.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/JwtManager.java new file mode 100644 index 000000000..4bda7665d --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/JwtManager.java @@ -0,0 +1,121 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.token; + +import com.sun.jersey.core.util.Base64; +import com.webank.wedatasphere.dss.apiservice.core.config.ApiServiceConfiguration; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceToken; +import com.webank.wedatasphere.linkis.common.conf.CommonVars$; +import io.jsonwebtoken.Claims; +import io.jsonwebtoken.ExpiredJwtException; +import io.jsonwebtoken.Jws; +import io.jsonwebtoken.Jwts; +import io.jsonwebtoken.MalformedJwtException; +import io.jsonwebtoken.SignatureAlgorithm; +import io.jsonwebtoken.SignatureException; +import io.jsonwebtoken.UnsupportedJwtException; +import io.jsonwebtoken.impl.crypto.MacProvider; +import org.mortbay.log.Log; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import javax.crypto.SecretKey; +import javax.crypto.spec.SecretKeySpec; +import java.time.Duration; +import java.time.Instant; +import java.time.temporal.TemporalAmount; +import java.util.Date; +import java.util.HashMap; +import java.util.Map; +import java.util.Properties; + + +public class JwtManager { + private static final Logger LOG = LoggerFactory.getLogger(JwtManager.class); + + private static final String CLAIM_ROLE = "role"; + + private static final SignatureAlgorithm SIGNATURE_ALGORITHM = SignatureAlgorithm.HS256; + private static final SecretKey SECRET_KEY = MacProvider.generateKey(SIGNATURE_ALGORITHM); + private static final TemporalAmount TOKEN_VALIDITY = getTokenHour(); + private static final String JWT_SECERT = ApiServiceConfiguration.DSS_API_TOKEN_SECRET_ID.getValue(); + + public final static TemporalAmount getTokenHour() { + Properties conf = CommonVars$.MODULE$.properties(); + if (null != conf) { + Object hour = conf.get("ujes.token.valid.hour"); + if (null == hour) { + return Duration.ofHours(8L); + } else { + return Duration.ofHours(Long.parseLong(hour.toString())); + } + } else { + return Duration.ofDays(365); + } + + } + + public static SecretKey generalKey() { + byte[] encodedKey = Base64.decode(JWT_SECERT); + SecretKey key = new SecretKeySpec(encodedKey, 0, encodedKey.length, "AES"); + Log.info("token key str:"+key.toString()); + return key; + } + + /** + * Builds a JWT with the given subject and role and returns it as a JWS signed compact String. + */ + public static String createToken(final String applyUser, ApiServiceToken apiServiceToken, Long duration) { + final String role = "developer"; + final Instant now = Instant.now(); + final Date expiryDate = Date.from(now.plus(Duration.ofDays(duration))); + Map tokenDetailMap = new HashMap(); + tokenDetailMap.put("tokenDetail", apiServiceToken); + return Jwts.builder() + .setSubject(applyUser) + .claim(CLAIM_ROLE, role) + .setClaims(tokenDetailMap) + .setExpiration(expiryDate) + .setIssuedAt(Date.from(now)) + .signWith(SIGNATURE_ALGORITHM, generalKey()) + .compact(); + } + + /** + * Parses the given JWS signed compact JWT, returning the claims. + * If this method returns without throwing an exception, the token can be trusted. + */ + public static ApiServiceToken parseToken(final String compactToken) + throws ExpiredJwtException, + UnsupportedJwtException, + MalformedJwtException, + SignatureException, + IllegalArgumentException { + + Jws jws = Jwts.parser().setSigningKey(generalKey()).parseClaimsJws(compactToken); + + Map tokenDetail = (Map) jws.getBody().get("tokenDetail"); + ApiServiceToken apiServiceToken = new ApiServiceToken(); + apiServiceToken.setApiServiceId(Long.valueOf(tokenDetail.get("apiServiceId").toString())); + apiServiceToken.setApplyTime(null); + apiServiceToken.setPublisher(tokenDetail.get("publisher").toString()); + apiServiceToken.setApplyUser(tokenDetail.get("applyUser").toString()); + + return apiServiceToken; + } + + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/TokenAuth.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/TokenAuth.java new file mode 100644 index 000000000..b7385ea8c --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/TokenAuth.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.token; + +import com.webank.wedatasphere.dss.apiservice.core.constant.SaveTokenEnum; +import com.webank.wedatasphere.dss.apiservice.core.exception.ApiServiceTokenException; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApprovalVo; +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; + +import java.util.List; + + +public interface TokenAuth { + + + /** + * batch save token to db + * @param tokenManagerVos + * @return + */ + SaveTokenEnum saveTokensToDb(List tokenManagerVos, String approvalNo) throws ApiServiceTokenException; + + List genTokenRecord(ApprovalVo approvalVo); +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/TokenStatusCheckerTask.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/TokenStatusCheckerTask.java new file mode 100644 index 000000000..95203fc07 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/token/TokenStatusCheckerTask.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.token; + +import com.webank.wedatasphere.dss.apiservice.core.dao.ApiServiceTokenManagerDao; +import com.webank.wedatasphere.dss.apiservice.core.util.DateUtil; +import com.webank.wedatasphere.dss.apiservice.core.vo.TokenManagerVo; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.scheduling.annotation.EnableScheduling; +import org.springframework.scheduling.annotation.Scheduled; +import org.springframework.stereotype.Component; + +import java.util.Calendar; +import java.util.Date; +import java.util.List; + +@Component +@EnableScheduling +public class TokenStatusCheckerTask { + private static final Logger LOG = LoggerFactory.getLogger(TokenStatusCheckerTask.class); + + @Autowired + ApiServiceTokenManagerDao atmd; + + @Scheduled(cron = "0 0/5 * * * ?") + public void doTokenStatusCheckTask() { + // 查询启用状态的token + List tokenManagerVos = atmd.queryTokenByStatus(1); + if (null != tokenManagerVos) { + for (TokenManagerVo tmv : tokenManagerVos) { + Date applyTime = tmv.getApplyTime(); + Date nowTime = Calendar.getInstance().getTime(); + Calendar cal = Calendar.getInstance(); + cal.setTime(applyTime); + cal.add(Calendar.DATE, tmv.getDuration().intValue()); + Date endTime = cal.getTime(); + if (endTime.compareTo(nowTime) < 0) { + LOG.warn("token id:" + tmv.getId() + " 已经过期！"); + atmd.disableTokenStatus(tmv.getId()); + } + } + } + } + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/ApiUtils.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/ApiUtils.java new file mode 100644 index 000000000..e7baaeac0 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/ApiUtils.java @@ -0,0 +1,122 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.util; + +import com.webank.wedatasphere.dss.apiservice.core.exception.*; +import com.webank.wedatasphere.dss.apiservice.core.vo.MessageVo; +import com.webank.wedatasphere.linkis.common.exception.WarnException; +import com.webank.wedatasphere.linkis.server.Message; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.validation.ConstraintViolationException; +import javax.ws.rs.core.Response; + + +public class ApiUtils { + + private static final Logger LOG = LoggerFactory.getLogger(ApiUtils.class); + + /** + * @param tryOperation operate function + * @param failMessage message + */ + public static Response doAndResponse(TryOperation tryOperation, String method, String failMessage) { + try { + Message message = tryOperation.operateAndGetMessage(); + return Message.messageToResponse(setMethod(message, method)); + } catch (ConstraintViolationException e) { + LOG.error("api error, method: " + method, e); + return new BeanValidationExceptionMapper().toResponse(e); + } catch (WarnException e) { + LOG.error("api error, method: " + method, e); + return Message.messageToResponse(setMethod(Message.warn(e.getMessage()), method)); + } catch (AssertException e) { + LOG.error("api error, method: " + method, e); + return Message.messageToResponse(setMethod(Message.error(e.getMessage()), method)); + + } catch (ApiExecuteException e) { + LOG.error("api error, method: " + method, e); + return Message.messageToResponse(setMethod(Message.error(e.getMessage()), method)); + } + catch (ApiServiceQueryException e) { + LOG.error("api error, method: " + method, e); + return Message.messageToResponse(setMethod(Message.error(e.getMessage()), method)); + } + catch (Exception e) { + LOG.error("api error, method: " + method, e); + return Message.messageToResponse(setMethod(Message.error(failMessage, e), method)); + } + } + + /** + * @param tryOperation operate function + */ + public static Response doAndResponse(Operation tryOperation) { + Object msg = null; + try { + msg = tryOperation.operateAndGetMessage(); + return Response.ok(msg).build(); + } catch (ConstraintViolationException e) { + LOG.error("api error ", e); + return new BeanValidationExceptionMapper().toResponse(e); + } catch (WarnException e) { + LOG.error("api error ", e); + return Response.ok(setMsg("系统异常")).build(); + } catch (AssertException e) { + LOG.error("api error ", e); + return Response.ok(setMsg(e.getMessage())).build(); + }catch (ApiServiceRuntimeException e){ + LOG.error("api error ", e); + return Response.ok(setMsg(e.getMessage())).build(); + } + catch (Exception e) { + LOG.error("api error ", e); + return Response.ok(setMsg(String.valueOf(e.getCause()))).build(); + } + } + + public static Object setMsg(String message) { + MessageVo messageVo = new MessageVo(); + messageVo.setMessage(message); + + return messageVo; + } + + private static Message setMethod(Message message, String method) { + message.setMethod(method); + return message; + } + + @FunctionalInterface + public interface TryOperation { + + /** + * Operate method + */ + Message operateAndGetMessage() throws Exception; + } + + @FunctionalInterface + public interface Operation { + + /** + * Operate method + */ + Object operateAndGetMessage() throws Exception; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/AssertUtil.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/AssertUtil.java new file mode 100644 index 000000000..2ade7601d --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/AssertUtil.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.util; + + +import com.webank.wedatasphere.dss.apiservice.core.exception.AssertException; + + +public class AssertUtil { + + public static void isTrue(boolean b, String message) { + if (!b) { + throw new AssertException(message); + } + } + + public static void isFalse(boolean b, String message) { + if (b) { + throw new AssertException(message); + } + } + + public static void notEmpty(String str, String message) { + if (str == null || str.isEmpty()) { + throw new AssertException(message); + } + } + + public static void notNull(Object str, String message) { + if (str == null) { + throw new AssertException(message); + } + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/DateUtil.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/DateUtil.java new file mode 100644 index 000000000..5d5dcaa60 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/DateUtil.java @@ -0,0 +1,353 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.util; + +import java.sql.Timestamp; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.Calendar; +import java.util.Date; + + +public class DateUtil { + + /** + * 英文简写（默认）如：2010-12-01 + */ + public static String FORMAT_MONTH = "yyyy-MM"; + /** + * 英文简写（默认）如：2010-12-01 + */ + public static String FORMAT_SHORT = "yyyy-MM-dd"; + + public static String FORMAT_SHORT_REPORT = "yyyy/MM/dd"; + /** + * 英文全称 如：2010-12-01 23:15:06 + */ + public static String FORMAT_LONG = "yyyy-MM-dd HH:mm:ss"; + public static String FORMAT_WITHOUT_SECOND = "yyyy-MM-dd HH:mm"; + + public static String FORMAT_MINUTE_REPORT = "yyyy/MM/dd HH:mm"; + public static String FORMAT_MINUTE = "yyyy-MM-dd HH:mm"; + /** + * 精确到毫秒的完整时间 如：yyyy-MM-dd HH:mm:ss.S + */ + public static String FORMAT_FULL = "yyyy-MM-dd HH:mm:ss.S"; + /** + * 中文简写 如：2010年12月01日 + */ + public static String FORMAT_SHORT_CN = "yyyy年MM月dd日"; + /** + * 中文全称 如：2010年12月01日 23时15分06秒 + */ + public static String FORMAT_LONG_CN = "yyyy年MM月dd日 HH时mm分ss秒"; + /** + * 精确到毫秒的完整中文时间 + */ + public static String FORMAT_FULL_CN = "yyyy年MM月dd日 HH时mm分ss秒SSS毫秒"; + + /** + * 获得默认的 date pattern + */ + public static String getDatePattern() { + return FORMAT_LONG; + } + + /** + * 根据预设格式返回当前日期 + * + * @return + */ + public static String getNow() { + return format(new Date()); + } + + /** + * 根据用户格式返回当前日期 + * + * @param format + * @return + */ + public static String getNow(String format) { + return format(new Date(), format); + } + + /** + * 使用预设格式格式化日期 + * + * @param date + * @return + */ + public static String format(Date date) { + return format(date, getDatePattern()); + } + + /** + * 使用用户格式格式化日期 + * + * @param date 日期 + * @param pattern 日期格式 + * @return + */ + public static String format(Date date, String pattern) { + String returnValue = ""; + if (date != null) { + SimpleDateFormat df = new SimpleDateFormat(pattern); + returnValue = df.format(date); + } + return (returnValue); + } + + /** + * 使用预设格式提取字符串日期 + * + * @param strDate 日期字符串 + * @return + */ + public static Date parse(String strDate) { + return parse(strDate, getDatePattern()); + } + + /** + * 使用用户格式提取字符串日期 + * + * @param strDate 日期字符串 + * @param pattern 日期格式 + * @return + */ + public static Date parse(String strDate, String pattern) { + SimpleDateFormat df = new SimpleDateFormat(pattern); + try { + return df.parse(strDate); + } catch (ParseException e) { + e.printStackTrace(); + return null; + } + } + + /** + * 在日期上增加数个整月 + * + * @param date 日期 + * @param n 要增加的月数 + * @return + */ + public static Date addMonth(Date date, int n) { + Calendar cal = Calendar.getInstance(); + cal.setTime(date); + cal.add(Calendar.MONTH, n); + return cal.getTime(); + } + + /** + * 在日期上增加天数 + * + * @param date 日期 + * @param n 要增加的天数 + * @return + */ + public static Date addDay(Date date, int n) { + Calendar cal = Calendar.getInstance(); + cal.setTime(date); + cal.add(Calendar.DATE, n); + return cal.getTime(); + } + + /** + * 获取时间戳 + */ + public static String getTimeString() { + SimpleDateFormat df = new SimpleDateFormat(FORMAT_FULL); + Calendar calendar = Calendar.getInstance(); + return df.format(calendar.getTime()); + } + + /** + * 获取日期年份 + * + * @param date 日期 + * @return + */ + public static String getYear(Date date) { + return format(date).substring(0, 4); + } + + /** + * 按默认格式的字符串距离今天的天数 + * + * @param date 日期字符串 + * @return + */ + public static int countDays(String date) { + long t = Calendar.getInstance().getTime().getTime(); + Calendar c = Calendar.getInstance(); + c.setTime(parse(date)); + long t1 = c.getTime().getTime(); + return (int) (t / 1000 - t1 / 1000) / 3600 / 24; + } + + /** + * 按用户格式字符串距离今天的天数 + * + * @param date 日期字符串 + * @param format 日期格式 + * @return + */ + public static int countDaysAbs(String date, String format) { + long t = Calendar.getInstance().getTime().getTime(); + Calendar c = Calendar.getInstance(); + c.setTime(parse(date, format)); + long t1 = c.getTime().getTime(); + return (int) Math.abs(t / 1000 - t1 / 1000) / 3600 / 24; + } + + /** + * 按用户格式字符串距离今天的天数 + * + * @param date 日期字符串 + * @param format 日期格式 + * @return + */ + public static int countDays(String date, String format) { + long t = Calendar.getInstance().getTime().getTime(); + Calendar c = Calendar.getInstance(); + c.setTime(parse(date, format)); + long t1 = c.getTime().getTime(); + return (int) (t / 1000 - t1 / 1000) / 3600 / 24; + } + + public static String timeStrComplete(String timeStr) { + if (null == timeStr || "".equals(timeStr)) { + return null; + } + String str = timeStr; + str = str.replaceAll("[0-9]", "0"); + + String tempStr = "0000-00-00 00:00:00"; + String tempSplitStr = "1970-01-01 00:00:00"; + int index = tempStr.indexOf(str); + if (0 == index) { + /** + * 说明符合日期格式 截取tempStr后面的部分与timeStr拼接构成时间 + */ + String lastStr = tempSplitStr.substring(timeStr.length()); + return timeStr + lastStr; + } + return null; + } + + public static Timestamp str2Time(String timeStr) { + if (null == timeStr || "".equals(timeStr)) { + return null; + } + String str = timeStr; + str = str.replaceAll("[0-9]", "0"); + + String tempStr = "0000-00-00 00:00:00"; + String tempSplitStr = "1970-01-01 00:00:00"; + int index = tempStr.indexOf(str); + if (0 == index) { + /** + * 说明符合日期格式 截取tempStr后面的部分与timeStr拼接构成时间 + */ + String lastStr = tempSplitStr.substring(timeStr.length()); + return Timestamp.valueOf(timeStr + lastStr); + } + + return null; + } + + public static Timestamp str2TimeMax(String timeStr) { + if (null == timeStr || "".equals(timeStr)) { + return null; + } + String str = timeStr; + str = str.replaceAll("[0-9]", "0"); + + String tempStr = "0000-00-00 00:00:00"; + String tempSplitStr = "1970-01-01 23:59:59"; + int index = tempStr.indexOf(str); + if (0 == index) { + /** + * 说明符合日期格式 截取tempStr后面的部分与timeStr拼接构成时间 + */ + String lastStr = tempSplitStr.substring(timeStr.length()); + return Timestamp.valueOf(timeStr + lastStr); + } + + return null; + } + + public static Date str2DateTime(String timeStr) { + if (null == timeStr || "".equals(timeStr)) { + return null; + } + String str = timeStr; + str = str.replaceAll("[0-9]", "0"); + + String tempStr = "0000-00-00 00:00:00"; + String tempSplitStr = "1970-01-01 00:00:00"; + int index = tempStr.indexOf(str); + if (0 == index) { + /** + * 说明符合日期格式 截取tempStr后面的部分与timeStr拼接构成时间 + */ + String lastStr = tempSplitStr.substring(timeStr.length()); + return parse(timeStr + lastStr); + } + + return null; + } + + public static Date str2DateTimeMax(String timeStr) { + if (null == timeStr || "".equals(timeStr)) { + return null; + } + String str = timeStr; + str = str.replaceAll("[0-9]", "0"); + + String tempStr = "0000-00-00 00:00:00"; + String tempSplitStr = "1970-01-01 23:59:59"; + int index = tempStr.indexOf(str); + if (0 == index) { + /** + * 说明符合日期格式 截取tempStr后面的部分与timeStr拼接构成时间 + */ + String lastStr = tempSplitStr.substring(timeStr.length()); + return parse(timeStr + lastStr); + } + + return null; + } + + /** + * 字符串数字转化为double + * + * @param string + * @return + */ + public static double str2double(String string) { + if (null == string || "".equals(string)) { + return 0; + } + try { + return Double.valueOf(string); + } catch (NumberFormatException e) { + return 0; + } + } + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/HttpClientUtil.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/HttpClientUtil.java new file mode 100644 index 000000000..e01f5a7bc --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/HttpClientUtil.java @@ -0,0 +1,477 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.util; + +import org.apache.http.Consts; +import org.apache.http.HttpEntity; +import org.apache.http.HttpResponse; +import org.apache.http.NameValuePair; +import org.apache.http.client.ClientProtocolException; +import org.apache.http.client.config.RequestConfig; +import org.apache.http.client.entity.UrlEncodedFormEntity; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.entity.ContentType; +import org.apache.http.entity.StringEntity; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; +import org.apache.http.message.BasicNameValuePair; +import org.apache.http.util.EntityUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.net.ssl.TrustManager; +import javax.net.ssl.X509TrustManager; +import java.io.IOException; +import java.io.UnsupportedEncodingException; +import java.net.SocketTimeoutException; +import java.net.URLEncoder; +import java.security.cert.CertificateException; +import java.text.SimpleDateFormat; +import java.util.*; +import java.util.Map.Entry; + +//import com.alibaba.fastjson.JSONArray; +//import com.alibaba.fastjson.JSONObject; +//import com.alibaba.fastjson.serializer.SerializerFeature; + +@SuppressWarnings("all") +public final class HttpClientUtil { + private final static Logger logger = LoggerFactory.getLogger(HttpClientUtil.class); + public final static int connectTimeout = 5000; + private static PoolingHttpClientConnectionManager connManager = null; + private static CloseableHttpClient httpclient = null; + + /** + * 重写验证方法，取消检测ssl + */ + private static TrustManager trustAllManager = new X509TrustManager() { + @Override + public void checkClientTrusted(java.security.cert.X509Certificate[] arg0, String arg1) + throws CertificateException { + } + @Override + public void checkServerTrusted(java.security.cert.X509Certificate[] arg0, String arg1) + throws CertificateException { + } + @Override + public java.security.cert.X509Certificate[] getAcceptedIssuers() { + return null; + } + + }; + + static { + httpclient = HttpClients.createDefault(); + } + + + /** + * + * @param url + * @param timeout + * @param headerMap + * @param paramsList + * @param encoding + * @return + */ + public static String postForm(String url, int timeout, Map headerMap, List paramsList, String encoding){ + HttpPost post = new HttpPost(url); + try { + if(headerMap != null){ + for(Entry entry : headerMap.entrySet()){ + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + //post.setHeader("Content-type", "application/json"); + RequestConfig requestConfig = RequestConfig.custom() + .setSocketTimeout(timeout) + .setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout) + .setExpectContinueEnabled(false).build(); + post.setConfig(requestConfig); + + post.setEntity(new UrlEncodedFormEntity(paramsList, encoding)); + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if(entity != null){ + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if(entity != null){ + entity.getContent().close(); + } + } + } finally { + if(response != null){ + response.close(); + } + } + } catch (Exception e) { + throw new RuntimeException("invoke http post error!",e); + } finally { + post.releaseConnection(); + } + return ""; + } + + /** + * 调用saltapi时 + * + * @author: XIEJIAN948@pingan.com.cn + */ + public static String postJsonBody(String url, int timeout, Map headerMap, + String paraData, String encoding) { + + logger.info("successfully start post Json Body url{} ", url); + HttpPost post = new HttpPost(url); + try { + if (headerMap != null) { + for (Entry entry : headerMap.entrySet()) { + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout).setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout).setExpectContinueEnabled(false).build(); + StringEntity jsonEntity = new StringEntity(paraData, ContentType.APPLICATION_JSON); + post.setConfig(requestConfig); + post.setEntity(jsonEntity); + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + } catch (UnsupportedEncodingException e) { + logger.error("UnsupportedEncodingException", e); + throw new RuntimeException("failed post json return blank!"); + } catch (Exception e) { + logger.error("Exception", e); + throw new RuntimeException("failed post json return blank!"); + } finally { + post.releaseConnection(); + } + logger.info("successfully end post Json Body url{} ", url); + return ""; + } + + @SuppressWarnings("deprecation") + public static String invokeGet(String url, Map params, String encode, int connectTimeout, + int soTimeout) { + String responseString = null; + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(connectTimeout) + .setConnectTimeout(connectTimeout).setConnectionRequestTimeout(connectTimeout).build(); + + StringBuilder sb = new StringBuilder(); + sb.append(url); + int i = 0; + if (params != null) { + for (Entry entry : params.entrySet()) { + if (i == 0 && !url.contains("?")) { + sb.append("?"); + } else { + sb.append("&"); + } + sb.append(entry.getKey()); + sb.append("="); + String value = entry.getValue(); + try { + sb.append(URLEncoder.encode(value, "UTF-8")); + } catch (UnsupportedEncodingException e) { + logger.warn("encode http get params error, value is " + value, e); + sb.append(URLEncoder.encode(value)); + } + i++; + } + } + HttpGet get = new HttpGet(sb.toString()); + get.setConfig(requestConfig); + try { + CloseableHttpResponse response = httpclient.execute(get); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + responseString = EntityUtils.toString(entity, encode); + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } catch (Exception e) { + logger.error(String.format("[HttpUtils Get]get response error, url:%s", sb.toString()), e); + return responseString; + } finally { + if (response != null) { + response.close(); + } + } + // System.out.println(String.format("[HttpUtils Get]Debug url:%s , + // response string %s:", sb.toString(), responseString)); + } catch (SocketTimeoutException e) { + logger.error(String.format("[HttpUtils Get]invoke get timout error, url:%s", sb.toString()), e); + return responseString; + } catch (Exception e) { + logger.error(String.format("[HttpUtils Get]invoke get error, url:%s", sb.toString()), e); + } finally { + get.releaseConnection(); + } + return responseString; + } + + /** + * HTTPS请求，默认超时为5S + * + * @param reqURL + * @param params + * @return + */ + public static String connectPostHttps(String reqURL, Map params) { + + String responseContent = null; + HttpPost httpPost = new HttpPost(reqURL); + try { + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(connectTimeout) + .setConnectTimeout(connectTimeout).setConnectionRequestTimeout(connectTimeout).build(); + List formParams = new ArrayList(); + httpPost.setEntity(new UrlEncodedFormEntity(formParams, Consts.UTF_8)); + httpPost.setConfig(requestConfig); + // 绑定到请求 Entry + for (Entry entry : params.entrySet()) { + formParams.add(new BasicNameValuePair(entry.getKey(), entry.getValue())); + } + CloseableHttpResponse response = httpclient.execute(httpPost); + try { + // 执行POST请求 + HttpEntity entity = response.getEntity(); // 获取响应实体 + try { + if (null != entity) { + responseContent = EntityUtils.toString(entity, Consts.UTF_8); + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + logger.info("requestURI : " + httpPost.getURI() + ", responseContent: " + responseContent); + } catch (ClientProtocolException e) { + logger.error("ClientProtocolException", e); + } catch (IOException e) { + logger.error("IOException", e); + } finally { + httpPost.releaseConnection(); + } + return responseContent; + + } + + class Test { + String v; + String k; + + public String getV() { + return v; + } + + public void setV(String v) { + this.v = v; + } + + public String getK() { + return k; + } + + public void setK(String k) { + this.k = k; + } + + } + + // 随机4位数 + public static String getRandomValue() { + String str = "0123456789"; + StringBuilder sb = new StringBuilder(4); + for (int i = 0; i < 4; i++) { + char ch = str.charAt(new Random().nextInt(str.length())); + sb.append(ch); + } + return sb.toString(); + } + + // 当前时间到秒 + public static String getTimestamp() { + + Date date = new Date(); + String timestamp = String.valueOf(date.getTime() / 1000); + return timestamp; + } + + // 当前时间到秒 + public static String getNowDate() { + Date date = new Date(); + SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMddHHmmss"); + return sdf.format(date); + } + + /** + * 调用saltapi时 + * + * @author: XIEJIAN948@pingan.com.cn + */ + public static String postJsonBody2(String url, int timeout, Map headerMap, + List paramsList, String encoding) { + logger.info("successfully start post Json Body url{} ", url); + HttpPost post = new HttpPost(url); + try { + if (headerMap != null) { + for (Entry entry : headerMap.entrySet()) { + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout).setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout).setExpectContinueEnabled(false).build(); + post.setConfig(requestConfig); + if (paramsList.size() > 0) { + UrlEncodedFormEntity entity = new UrlEncodedFormEntity(paramsList, encoding); + post.setEntity(entity); + } + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + } catch (UnsupportedEncodingException e) { + logger.error("UnsupportedEncodingException", e); + throw new RuntimeException("failed post json return blank!"); + } catch (Exception e) { + logger.error("Exception", e); + throw new RuntimeException("failed post json return blank!"); + } finally { + post.releaseConnection(); + } + logger.info("successfully end post Json Body url{} ", url); + return ""; + } + + /** + * 调用saltapi时 + * + * @author: XIEJIAN948@pingan.com.cn + */ + public static String postJsonBody3(String url, int timeout, Map headerMap, + Map paramsList, String encoding) { + HttpPost post = new HttpPost(url); + try { + if (headerMap != null) { + for (Entry entry : headerMap.entrySet()) { + post.setHeader(entry.getKey(), entry.getValue().toString()); + } + } + RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(timeout).setConnectTimeout(timeout) + .setConnectionRequestTimeout(timeout).setExpectContinueEnabled(false).build(); + post.setConfig(requestConfig); + if (paramsList.size() > 0) { + //JSONArray jsonArray = JSONArray.fromObject(paramsList); + //post.setEntity(new StringEntity(jsonArray.get(0).toString(), encoding)); + post.setEntity(new StringEntity(null, encoding)); + //logger.info("successfully start post Json Body url{},params ", url,jsonArray.get(0).toString()); + logger.info("successfully start post Json Body url{},params ", url,null); + } + CloseableHttpResponse response = httpclient.execute(post); + try { + HttpEntity entity = response.getEntity(); + try { + if (entity != null) { + String str = EntityUtils.toString(entity, encoding); + return str; + } + } finally { + if (entity != null) { + entity.getContent().close(); + } + } + } finally { + if (response != null) { + response.close(); + } + } + } catch (UnsupportedEncodingException e) { + logger.error("UnsupportedEncodingException", e); + throw new RuntimeException("failed post json return blank!"); + } catch (Exception e) { + logger.error("Exception", e); + throw new RuntimeException("failed post json return blank!"); + } finally { + post.releaseConnection(); + } + logger.info("successfully end post Json Body url{} ", url); + return ""; + } + + public static String executeGet(String url) + { + String rtnStr = ""; + HttpGet httpGet = new HttpGet(url); + try { + HttpResponse httpResponse = httpclient.execute(httpGet); + //获得返回的结果 + rtnStr = EntityUtils.toString(httpResponse.getEntity()); + } catch (IOException e) { + e.printStackTrace(); + } finally { + httpGet.releaseConnection(); + } + return rtnStr; + } + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/ModelMapperUtil.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/ModelMapperUtil.java new file mode 100644 index 000000000..89b49d23d --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/ModelMapperUtil.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.util; + +import org.modelmapper.ModelMapper; +import org.modelmapper.convention.MatchingStrategies; + +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + + +public class ModelMapperUtil { + private static ModelMapper modelMapper = new ModelMapper(); + + public ModelMapperUtil() { + } + + public static D strictMap(Object source, Class destinationType) { + return modelMapper.map(source, destinationType); + } + + public static List strictMapList(Object source, Class componentType) { + List list = new ArrayList(); + List objectList = (List)source; + Iterator var4 = objectList.iterator(); + + while(var4.hasNext()) { + Object obj = var4.next(); + list.add(modelMapper.map(obj, componentType)); + } + + return list; + } + + static { + modelMapper.getConfiguration().setMatchingStrategy(MatchingStrategies.STRICT); + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/SQLCheckUtil.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/SQLCheckUtil.java new file mode 100644 index 000000000..54b71be63 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/SQLCheckUtil.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.util; + +import java.util.regex.Matcher; +import java.util.regex.Pattern; + + +public class SQLCheckUtil { + + // sql 注入检查 + private static String injectionReg = "(?:--)|" + + "(\\b(select|update|union|and|or|delete|insert|trancate|char|substr|ascii|declare|exec|count|master|into|drop|execute)\\b)"; + private static Pattern paramInjectionPattern = Pattern.compile(injectionReg, Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + + /** + * 1. -- 注释 + * 2. select * from table -- 注释 + * 但是不包含 -- '注释' + * */ + private static String sqlCommentReg = "\\-\\-([^\\'\\r\\n]{0,}(\\'[^\\'\\r\\n]{0,}\\'){0,1}[^\\'\\r\\n]{0,}){0,}"; + private static Pattern sqlCommentPattern = Pattern.compile(sqlCommentReg, Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + + public static boolean doParamInjectionCheck(String str) { + Matcher matcher = paramInjectionPattern.matcher(str); + if(matcher.find()) { + return true; + } + return false; + } + + public static String sqlCommentReplace(String sql) { + String newSql = sql.replaceAll(sqlCommentReg, ""); + return newSql; + } + + public static void main(String[] args) { + String sql = "--注释\n" + + "-- 注释\n" + + "select * from tb --注释\n"; + String newSql = sqlCommentReplace(sql); + String[] selects = newSql.split("select"); + System.out.println(newSql); + System.out.println(selects.length); + } + +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/UUIDGenerator.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/UUIDGenerator.java new file mode 100644 index 000000000..5c0adc70b --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/util/UUIDGenerator.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.util; + +import java.util.UUID; + + +public class UUIDGenerator { + + /** + * 生成唯一标识uuid + * */ + public static String genUUID() { + String uuid = UUID.randomUUID().toString(); + return uuid; + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiAccessVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiAccessVo.java new file mode 100644 index 000000000..d75bac971 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiAccessVo.java @@ -0,0 +1,100 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.vo; + + +public class ApiAccessVo { + Long id; + + String user; + + String apiServiceName; + + Long apiServiceId; + + Long apiServiceVersionId; + + String apiPublisher; + + String accessTime; + + String proxyUser; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + + public String getProxyUser() { + return proxyUser; + } + + public void setProxyUser(String proxyUser) { + this.proxyUser = proxyUser; + } + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + public String getApiServiceName() { + return apiServiceName; + } + + public void setApiServiceName(String apiServiceName) { + this.apiServiceName = apiServiceName; + } + + public Long getApiServiceId() { + return apiServiceId; + } + + public void setApiServiceId(Long apiServiceId) { + this.apiServiceId = apiServiceId; + } + + public Long getApiServiceVersionId() { + return apiServiceVersionId; + } + + public void setApiServiceVersionId(Long apiServiceVersionId) { + this.apiServiceVersionId = apiServiceVersionId; + } + + public String getApiPublisher() { + return apiPublisher; + } + + public void setApiPublisher(String apiPublisher) { + this.apiPublisher = apiPublisher; + } + + public String getAccessTime() { + return accessTime; + } + + public void setAccessTime(String accessTime) { + this.accessTime = accessTime; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiServiceVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiServiceVo.java new file mode 100644 index 000000000..ff4858499 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiServiceVo.java @@ -0,0 +1,279 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.vo; + + +import java.util.Date; +import java.util.List; +import java.util.Map; + + +public class ApiServiceVo { + private Long id; + private String name; + private String aliasName; + private String path; + private int protocol; + private String method; + private String tag; + private String scope; + private String description; + private Integer status = 1; + private String type; + private String runType; + private Date createTime; + private Date modifyTime; + private String creator; + private String modifier; + private String scriptPath; + private ApprovalVo approvalVo; + + private Long latestVersionId; + private String userToken; + + private List params; + + private Map metadata; + private String content; + + private List versionVos; + + private Long workspaceId = 1L; + private Long targetServiceId; + private String comment; + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + + public Long getTargetServiceId() { + return targetServiceId; + } + + public void setTargetServiceId(Long targetServiceId) { + this.targetServiceId = targetServiceId; + } + + + + public ApprovalVo getApprovalVo() { + return approvalVo; + } + + public void setApprovalVo(ApprovalVo approvalVo) { + this.approvalVo = approvalVo; + } + + public Long getLatestVersionId() { + return latestVersionId; + } + + public void setLatestVersionId(Long latestVersionId) { + this.latestVersionId = latestVersionId; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getPath() { + return path; + } + + public void setPath(String path) { + this.path = path; + } + + public int getProtocol() { + return protocol; + } + + public void setProtocol(int protocol) { + this.protocol = protocol; + } + + public String getMethod() { + return method; + } + + public void setMethod(String method) { + this.method = method; + } + + public String getTag() { + return tag; + } + + public void setTag(String tag) { + this.tag = tag; + } + + public String getScope() { + return scope; + } + + public void setScope(String scope) { + this.scope = scope; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Integer getStatus() { + return status; + } + + public void setStatus(Integer status) { + this.status = status; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getModifyTime() { + return modifyTime; + } + + public void setModifyTime(Date modifyTime) { + this.modifyTime = modifyTime; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public String getModifier() { + return modifier; + } + + public void setModifier(String modifier) { + this.modifier = modifier; + } + + public String getContent() { + return content; + } + + public void setContent(String content) { + this.content = content; + } + + public String getScriptPath() { + return scriptPath; + } + + public void setScriptPath(String scriptPath) { + this.scriptPath = scriptPath; + } + + public Map getMetadata() { + return metadata; + } + + public void setMetadata(Map metadata) { + this.metadata = metadata; + } + + public List getVersionVos() { + return versionVos; + } + + public void setVersionVos(List versionVos) { + this.versionVos = versionVos; + } + + public List getParams() { + return params; + } + + public void setParams(List params) { + this.params = params; + } + + + public String getUserToken() { + return userToken; + } + + public void setUserToken(String userToken) { + this.userToken = userToken; + } + public String getRunType() { + return runType; + } + + public void setRunType(String runType) { + this.runType = runType; + } + + public String getAliasName() { + return aliasName; + } + + public void setAliasName(String aliasName) { + this.aliasName = aliasName; + } + +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiVersionVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiVersionVo.java new file mode 100644 index 000000000..cfd82fc86 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApiVersionVo.java @@ -0,0 +1,140 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.vo; + +import java.util.Date; +import java.util.List; + + +public class ApiVersionVo { + + private Long id; + private Long apiId; + private String version; + private String bmlResourceId; + private String bmlVersion; + private String source; + private String creator; + private Date createTime; + private String metadataInfo; + private String authId; + + + public int getStatus() { + return status; + } + + public void setStatus(int status) { + this.status = status; + } + + private int status; + + /** + * one version data corresponds to multiple param data + * */ + private List paramVos; + + public List getParamVos() { + return paramVos; + } + + public String getMetadataInfo() { + return metadataInfo; + } + + public String getAuthId() { + return authId; + } + + public void setAuthId(String authId) { + this.authId = authId; + } + + public void setMetadataInfo(String metadataInfo) { + this.metadataInfo = metadataInfo; + } + + public void setParamVos(List paramVos) { + this.paramVos = paramVos; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getApiId() { + return apiId; + } + + public void setApiId(Long apiId) { + this.apiId = apiId; + } + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public String getBmlResourceId() { + return bmlResourceId; + } + + public void setBmlResourceId(String bmlResourceId) { + this.bmlResourceId = bmlResourceId; + } + + public String getBmlVersion() { + return bmlVersion; + } + + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApprovalVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApprovalVo.java new file mode 100644 index 000000000..2b1eeec1b --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ApprovalVo.java @@ -0,0 +1,122 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.vo; + +import java.util.Date; + + +public class ApprovalVo { + Long id; + Long apiId; + Long apiVersionId; + String approvalName; + String applyUser; + String executeUser; + String creator; + Integer status; + Date createTime; + Date updateTime; + String approvalNo; + + public String getCreator() { + return creator; + } + + public Long getApiVersionId() { + return apiVersionId; + } + + public void setApiVersionId(Long apiVersionId) { + this.apiVersionId = apiVersionId; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getApiId() { + return apiId; + } + + public void setApiId(Long apiId) { + this.apiId = apiId; + } + + public String getApprovalName() { + return approvalName; + } + + public void setApprovalName(String approvalName) { + this.approvalName = approvalName; + } + + public String getApplyUser() { + return applyUser; + } + + public void setApplyUser(String applyUser) { + this.applyUser = applyUser; + } + + public String getExecuteUser() { + return executeUser; + } + + public void setExecuteUser(String executeUser) { + this.executeUser = executeUser; + } + + public Integer getStatus() { + return status; + } + + public void setStatus(Integer status) { + this.status = status; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getApprovalNo() { + return approvalNo; + } + + public void setApprovalNo(String approvalNo) { + this.approvalNo = approvalNo; + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/MessageExtVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/MessageExtVo.java new file mode 100644 index 000000000..84790e17f --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/MessageExtVo.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.vo; + +import java.util.LinkedHashMap; + + +public class MessageExtVo extends LinkedHashMap { + public static MessageExtVo build(MessageVo messageVo) { + MessageExtVo messageExtVo = new MessageExtVo(); + + messageExtVo.put("status", messageVo.getStatus()); + messageExtVo.put("message", messageVo.getMessage()); + messageExtVo.put("data", messageVo.getData()); + + return messageExtVo; + } + + public MessageExtVo setMessage(String message) { + put("message", message); + return this; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/MessageVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/MessageVo.java new file mode 100644 index 000000000..39e1dddc8 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/MessageVo.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.vo; + + +public class MessageVo { + private Integer status = 0; + + private String message = "OK"; + + private Object data; + + public MessageVo() { + } + + public MessageVo(Integer status, String message, Object data) { + this.status = status; + this.message = message; + this.data = data; + } + + public Integer getStatus() { + return status; + } + + public MessageVo setStatus(Integer status) { + this.status = status; + return this; + } + + public String getMessage() { + return message; + } + + public MessageVo setMessage(String message) { + this.message = message; + return this; + } + + public Object getData() { + return data; + } + + public MessageVo setData(Object data) { + this.data = data; + return this; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ParamVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ParamVo.java new file mode 100644 index 000000000..2dc6ccffd --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/ParamVo.java @@ -0,0 +1,102 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.vo; + + +public class ParamVo { + + private Long id; + private Long apiVersionId; + private String name; + private String type; + private Integer required = 1; + private String description; + private String displayName; + private String defaultValue; + private String details; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getApiVersionId() { + return apiVersionId; + } + + public void setApiVersionId(Long apiVersionId) { + this.apiVersionId = apiVersionId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public Integer getRequired() { + return required; + } + + public void setRequired(Integer required) { + this.required = required; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getDisplayName() { + return displayName; + } + + public void setDisplayName(String displayName) { + this.displayName = displayName; + } + + public String getDefaultValue() { + return defaultValue; + } + + public void setDefaultValue(String defaultValue) { + this.defaultValue = defaultValue; + } + + public String getDetails() { + return details; + } + + public void setDetails(String details) { + this.details = details; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/QueryParamVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/QueryParamVo.java new file mode 100644 index 000000000..cfc184eae --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/QueryParamVo.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.apiservice.core.vo; + + +public class QueryParamVo extends ParamVo { + + private String typeStr; + + private String requireStr; + + private String testValue; + + public String getTypeStr() { + return typeStr; + } + + public void setTypeStr(String typeStr) { + this.typeStr = typeStr; + } + + public String getRequireStr() { + return requireStr; + } + + public void setRequireStr(String requireStr) { + this.requireStr = requireStr; + } + + public String getTestValue() { + return testValue; + } + + public void setTestValue(String testValue) { + this.testValue = testValue; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/TokenManagerVo.java b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/TokenManagerVo.java new file mode 100644 index 000000000..13f239b5f --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/java/com/webank/wedatasphere/dss/apiservice/core/vo/TokenManagerVo.java @@ -0,0 +1,148 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.core.vo; + +import java.util.Date; + + +public class TokenManagerVo { + private Long id; + private Long apiId; + private Long apiVersionId; + public String publisher; + private String user; + private Date applyTime; + private Long duration; + private String reason; + private String ipWhitelist; + private Integer status; + private String caller; + private String accessLimit; + private String token; + private String applySource; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getApiId() { + return apiId; + } + + public void setApiId(Long apiId) { + this.apiId = apiId; + } + + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + public String getPublisher() { + return publisher; + } + + public void setPublisher(String publisher) { + this.publisher = publisher; + } + + public Date getApplyTime() { + return applyTime; + } + + public void setApplyTime(Date applyTime) { + this.applyTime = applyTime; + } + + public Long getDuration() { + return duration; + } + + public void setDuration(Long duration) { + this.duration = duration; + } + + public String getReason() { + return reason; + } + + public void setReason(String reason) { + this.reason = reason; + } + + public String getIpWhitelist() { + return ipWhitelist; + } + + public void setIpWhitelist(String ipWhitelist) { + this.ipWhitelist = ipWhitelist; + } + + public Integer getStatus() { + return status; + } + + public void setStatus(Integer status) { + this.status = status; + } + + public String getCaller() { + return caller; + } + + public void setCaller(String caller) { + this.caller = caller; + } + + public String getAccessLimit() { + return accessLimit; + } + + public void setAccessLimit(String accessLimit) { + this.accessLimit = accessLimit; + } + + public String getToken() { + return token; + } + + public void setToken(String token) { + this.token = token; + } + + public String getApplySource() { + return applySource; + } + + public void setApplySource(String applySource) { + this.applySource = applySource; + } + public Long getApiVersionId() { + return apiVersionId; + } + + public void setApiVersionId(Long apiVersionId) { + this.apiVersionId = apiVersionId; + } +} diff --git a/dss-apps/dss-apiservice-server/src/main/scala/com/webank/wedatasphere/dss/apiservice/DSSApiServiceServerApplication.scala b/dss-apps/dss-apiservice-server/src/main/scala/com/webank/wedatasphere/dss/apiservice/DSSApiServiceServerApplication.scala new file mode 100644 index 000000000..289d99905 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/main/scala/com/webank/wedatasphere/dss/apiservice/DSSApiServiceServerApplication.scala @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice + +import com.webank.wedatasphere.dss.common.utils.DSSMainHelper +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} + + +object DSSApiServiceServerApplication extends Logging { + + val userName: String = System.getProperty("user.name") + val hostName: String = Utils.getComputerName + + def main(args: Array[String]): Unit = { + val serviceName = System.getProperty("serviceName")//ProjectConf.SERVICE_NAME.getValue + DSSMainHelper.formatPropertyFiles(serviceName) + val allArgs = args ++ DSSMainHelper.getExtraSpringOptions + System.setProperty("hostName", hostName) + System.setProperty("userName", userName) + info(s"Ready to start $serviceName with args: ${allArgs.toList}.") + println(s"Test Ready to start $serviceName with args: ${allArgs.toList}.") + DataWorkCloudApplication.main(allArgs) + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/test/java/com/webank/wedatasphere/dss/apiservice/test/TestApiServiceDBOperation.java b/dss-apps/dss-apiservice-server/src/test/java/com/webank/wedatasphere/dss/apiservice/test/TestApiServiceDBOperation.java new file mode 100644 index 000000000..7c3290377 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/test/java/com/webank/wedatasphere/dss/apiservice/test/TestApiServiceDBOperation.java @@ -0,0 +1,157 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.apiservice.test; + +import com.ninja_squad.dbsetup.DbSetup; +import com.ninja_squad.dbsetup.DbSetupTracker; +import com.ninja_squad.dbsetup.Operations; +import com.ninja_squad.dbsetup.destination.DataSourceDestination; +import com.ninja_squad.dbsetup.operation.Operation; +import com.webank.wedatasphere.dss.apiservice.core.bo.ApiServiceToken; +import com.webank.wedatasphere.dss.apiservice.core.dao.ApiServiceAccessDao; +import com.webank.wedatasphere.dss.apiservice.core.dao.ApiServiceDao; +import com.webank.wedatasphere.dss.apiservice.core.service.ApiService; +import com.webank.wedatasphere.dss.apiservice.core.token.JwtManager; +import com.webank.wedatasphere.dss.apiservice.core.util.DateUtil; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiAccessVo; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiServiceVo; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApiVersionVo; +import com.webank.wedatasphere.dss.apiservice.core.vo.ApprovalVo; +import com.webank.wedatasphere.linkis.DataWorkCloudApplication; +import org.junit.Before; +import org.junit.Test; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.DisplayName; +import org.junit.runner.RunWith; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; + +import javax.sql.DataSource; +import java.util.Date; +import java.util.List; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.equalTo; + + + +@RunWith(SpringJUnit4ClassRunner.class) +//@MapperScan(annotationClass = Repository.class, basePackages = "com.webank.wedatasphere.dss.apiservice.dao" ) +@SpringBootTest(classes = {DataWorkCloudApplication.class}) +public class TestApiServiceDBOperation { + private static DbSetupTracker dbSetupTracker = new DbSetupTracker(); + @Autowired + ApiService apiService; + @Autowired + DataSource dataSource; + @Autowired + ApiServiceDao apiServiceDao; + + + @Autowired + ApiServiceAccessDao apiServiceAccessDao; + + @Before + public void setUp() { + Operation operation = Operations.sequenceOf( + Operations.deleteAllFrom("dss_apiservice_api", "dss_apiservice_param"), + Operations.insertInto("dss_apiservice_api") + .columns("name", "alias_name", "path", "protocol", "method", "tag", "scope", "description", "status", "type", "run_type", "create_time", "modify_time", "creator", "modifier", "script_path", "workspaceID", "api_comment") + .values("test01", "测试", "/test01", 0, "get", "new test", "shouquankejian", "miaoshu", "1", "sql", "spark", "2020-10-19 11:00:00", "2020-11-19 12:00:00", "allenlliu", "allenlliu", "123.sql", "180", "comment test") + .build(), + Operations.insertInto("dss_apiservice_param") + .columns("api_version_id", "name", "display_name", "type", "required", "default_value", "description", "details") + .values(1, "idcard", "身份证", "String", 1, "430124000000000", "test param", "身份标识") + .build() +// Operations.insertInto("dss_apiservice_access_info") +// .columns("api_version_id", "login_user", "display_name", "type", "required", "default_value", "description", "details").build(), + ); + DbSetup dbSetup = new DbSetup(new DataSourceDestination(dataSource), operation); + dbSetupTracker.launchIfNecessary(dbSetup); + } + + @DisplayName("ApiService库表验证") + @Test + public void testApiServiceDBOperate() { + dbSetupTracker.skipNextLaunch(); + + Integer actual = apiServiceDao.enableApi(0L); + ApiServiceVo apiServiceVo = apiServiceDao.queryByPath("/test01"); + Integer count = apiService.queryCountByName("test01"); + + Assertions.assertEquals(apiServiceVo.getName(), "test01"); + assertThat(actual, equalTo(0)); + assertThat(count, equalTo(1)); + } + + @DisplayName("ApiService备注更新测试") + @Test + public void testApiServiceCommentUpdate() { + dbSetupTracker.skipNextLaunch(); + + ApiServiceVo apiServiceVo01 = apiServiceDao.queryByPath("/test01"); + Integer actual = apiServiceDao.updateApiServiceComment(apiServiceVo01.getId(), "new comment"); + ApiServiceVo apiServiceVo02 = apiServiceDao.queryByPath("/test01"); + + + Assertions.assertEquals(apiServiceVo02.getComment(), "new comment"); + } + + @DisplayName("ApiService保存和版本验证") + @Test + public void testApiServiceVersion() { + dbSetupTracker.skipNextLaunch(); + + + ApiServiceVo apiServiceVo = apiServiceDao.queryByPath("/test01"); + apiServiceVo.setName("test02"); + apiServiceVo.setPath("/test02"); + //BML会上传失败，没有配置实际环境 + Assertions.assertThrows(NullPointerException.class, () -> { + apiService.save(apiServiceVo); + ; + }); + + } + + + + @DisplayName("数据服务访问记录验证") + @Test + public void testApiServiceAccessInfo() { + System.out.println("数据服务访问记录验证"); + ApiAccessVo apiAccessVo = new ApiAccessVo(); + apiAccessVo.setUser("allenlliu"); + apiAccessVo.setApiPublisher("testUser"); + apiAccessVo.setApiServiceName("testApi"); + apiAccessVo.setApiServiceId(102L); + apiAccessVo.setApiServiceVersionId(10L); + apiAccessVo.setProxyUser("hadoop"); + apiAccessVo.setAccessTime(DateUtil.getNow()); + apiServiceAccessDao.addAccessRecord(apiAccessVo); + + ApiAccessVo targetAccessVo = apiServiceAccessDao.queryByVersionId(10L); + Assertions.assertAll("accessInfos", + () -> Assertions.assertEquals(targetAccessVo.getProxyUser(), "hadoop"), + () -> Assertions.assertEquals(targetAccessVo.getApiServiceName(), "testApi"), + () -> Assertions.assertEquals(targetAccessVo.getApiPublisher(), "testUser") + + ); + + } +} \ No newline at end of file diff --git a/dss-apps/dss-apiservice-server/src/test/resources/linkis.properties b/dss-apps/dss-apiservice-server/src/test/resources/linkis.properties new file mode 100644 index 000000000..e68b6f963 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/test/resources/linkis.properties @@ -0,0 +1,57 @@ +# +# Copyright 2019 WeBank +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +# mysql 驱动: h2 +spring.datasource.driver-class-name=org.h2.Driver +# h2 内存数据库 库名: test +spring.datasource.url=jdbc:h2:mem:test +# 初始化数据表 +spring.datasource.schema=classpath:schema.sql +spring.datasource.username= +spring.datasource.password= +# 打印 SQL语句, Mapper所处的包 +logging.level.com.hawkingfoo.dao=debug + + + +wds.linkis.server.mybatis.mapperLocations=classpath:com/webank/wedatasphere/dss/apiservice/core/dao/mapper/*.xml +wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.apiservice.core +wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.apiservice.core.dao + +wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.apiservice.core.restful + +#sit +wds.linkis.server.version=v1 +wds.linkis.server.url= + +#test +wds.linkis.test.mode=false +wds.linkis.test.user= +wds.linkis.server.mybatis.datasource.url=jdbc:h2:mem:test +wds.linkis.server.mybatis.datasource.username= +***REMOVED*** + +#dsm +wds.linkis.server.dsm.admin.users= + +#bml +wds.linkis.gateway.url= +wds.linkis.gateway.ip= +wds.linkis.gateway.port= + +#用于执行的datasource配置 +wds.linkis.datasource.hikari.maximumPoolSize=100 +wds.linkis.datasource.hikari.minimumIdle=10 diff --git a/dss-apps/dss-apiservice-server/src/test/resources/schema.sql b/dss-apps/dss-apiservice-server/src/test/resources/schema.sql new file mode 100644 index 000000000..905024b93 --- /dev/null +++ b/dss-apps/dss-apiservice-server/src/test/resources/schema.sql @@ -0,0 +1,66 @@ +CREATE TABLE dss_apiservice_api ( + id bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + name varchar(200) NOT NULL COMMENT '服务名称', + alias_name varchar(200) NOT NULL COMMENT '服务中文名称', + path varchar(200) NOT NULL COMMENT '服务路径', + protocol int(11) NOT NULL COMMENT '协议: http, https', + method varchar(10) NOT NULL COMMENT '方法： post, put, delete', + tag varchar(200) DEFAULT NULL COMMENT '标签', + scope varchar(50) DEFAULT NULL COMMENT '范围', + description varchar(200) DEFAULT NULL COMMENT '服务描述', + status int(11) DEFAULT '0' COMMENT '服务状态，默认0是停止，1是运行中', + type varchar(50) DEFAULT NULL COMMENT '服务引擎类型', + run_type varchar(50) DEFAULT NULL COMMENT '脚本类型', + create_time timestamp NOT NULL COMMENT '创建时间', + modify_time timestamp NOT NULL COMMENT '修改时间', + creator varchar(50) DEFAULT NULL COMMENT '创建者', + modifier varchar(50) DEFAULT NULL COMMENT '修改者', + script_path varchar(200) NOT NULL COMMENT '脚本路径', + workspaceID int(11) NOT NULL COMMENT '工作空间ID', + api_comment varchar(1024) DEFAULT NULL COMMENT '服务备注', + PRIMARY KEY ( id ) +) COMMENT='服务api配置表'; + + +CREATE TABLE dss_apiservice_param ( + id bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + api_version_id bigint(20) NOT NULL COMMENT '服务api版本id', + name varchar(200) NOT NULL COMMENT '名称', + display_name varchar(50) DEFAULT NULL COMMENT '展示名', + type varchar(50) DEFAULT NULL COMMENT '类型', + required tinyint(1) DEFAULT '1' COMMENT '是否必须: 0否, 1是', + default_value varchar(200) DEFAULT NULL COMMENT '参数的默认值', + description varchar(200) DEFAULT NULL COMMENT '描述', + details varchar(500) DEFAULT NULL COMMENT '变量的详细说明', + PRIMARY KEY ( id ) +) COMMENT='apiservice 参数表'; + + +CREATE TABLE `dss_apiservice_api_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_id` bigint(20) NOT NULL COMMENT '服务的ID', + `version` varchar(50) NOT NULL COMMENT '服务对应的版本信息', + `bml_resource_id` varchar(50) NOT NULL COMMENT 'bml资源id', + `bml_version` varchar(20) NOT NULL COMMENT 'bml版本', + `source` varchar(200) DEFAULT NULL COMMENT '来源', + `creator` varchar(50) DEFAULT NULL COMMENT '创建者', + `create_time`timestamp NOT NULL COMMENT '创建时间', + `status` tinyint(1) default '1' COMMENT '0表示被禁用，1表示正在运行', + `metadata_info` varchar(5000) NOT NULL COMMENT '发布者库表信息', + `auth_id` varchar(200) NOT NULL COMMENT '用于与datamap交互的UUID', + `datamap_order_no` varchar(200) DEFAULT NULL COMMENT 'datamap审批单号码', + PRIMARY KEY(`id`) +) COMMENT='服务api版本表'; + + +CREATE TABLE `dss_apiservice_access_info` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `api_id` bigint(20) NOT NULL COMMENT '服务id', + `api_version_id` bigint(20) NOT NULL COMMENT '版本id', + `api_name` varchar(50) NOT NULL COMMENT '服务名称', + `login_user` varchar(50) NOT NULL COMMENT '提交用户', + `execute_user` varchar(50) DEFAULT NULL COMMENT '代理执行用户', + `api_publisher` varchar(50) NOT NULL COMMENT 'api创建者', + `access_time` timestamp NOT null COMMENT '访问时间', + PRIMARY KEY(`id`) +) COMMENT='apiservice 访问信息表'; \ No newline at end of file diff --git a/dss-apps/dss-datapipe-server/pom.xml b/dss-apps/dss-datapipe-server/pom.xml new file mode 100644 index 000000000..30f2dd9e5 --- /dev/null +++ b/dss-apps/dss-datapipe-server/pom.xml @@ -0,0 +1,285 @@ + + + + + 4.0.0 + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../pom.xml + + + dss-datapipe-server + + + UTF-8 + 2.16 + + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + asm + org.ow2.asm + + + hk2-api + org.glassfish.hk2 + + + jersey-common + org.glassfish.jersey.core + + + linkis-common + com.webank.wedatasphere.linkis + + + + + com.alibaba + fastjson + 1.2.70 + + + + org.glassfish.jersey.ext + jersey-bean-validation + ${jersey.version} + provided + + + javax.ws.rs-api + javax.ws.rs + + + hk2-locator + org.glassfish.hk2 + + + hk2-api + org.glassfish.hk2 + + + hibernate-validator + org.hibernate + + + jersey-server + org.glassfish.jersey.core + + + + + + com.webank.wedatasphere.linkis + linkis-bml-client + ${linkis.version} + + + commons-beanutils + commons-beanutils + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + com.webank.wedatasphere.linkis + linkis-computation-client + ${linkis.version} + + + commons-beanutils + commons-beanutils + + + linkis-common + com.webank.wedatasphere.linkis + + + + + org.postgresql + postgresql + 42.2.12 + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + + + linkis-common + com.webank.wedatasphere.linkis + + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + archaius-core + com.netflix.archaius + + + slf4j-api + org.slf4j + + + spring-cloud-starter + org.springframework.cloud + + + spring-web + org.springframework + + + + + hk2-api + org.glassfish.hk2 + 2.4.0-b34 + + + org.modelmapper + modelmapper + 0.7.5 + + + + io.jsonwebtoken + jjwt + 0.6.0 + + + jackson-databind + com.fasterxml.jackson.core + + + + + + junit + junit + 4.12 + test + + + org.junit.platform + junit-platform-launcher + 1.5.2 + test + + + xstream + com.thoughtworks.xstream + 1.4.11.1 + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + ${basedir}/src/main/resources + + + ${project.artifactId}-${project.version} + + \ No newline at end of file diff --git a/dss-apps/dss-datapipe-server/src/main/assembly/distribution.xml b/dss-apps/dss-datapipe-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..d6c12b0fb --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/assembly/distribution.xml @@ -0,0 +1,44 @@ + + + + dss-apiService-server + + dir + + true + dss-datapipe-server + + + + + + lib + true + true + false + true + true + + + + + + + diff --git a/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/config/DataPipeServiceConfiguration.java b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/config/DataPipeServiceConfiguration.java new file mode 100644 index 000000000..16185ebf4 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/config/DataPipeServiceConfiguration.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.config; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public class DataPipeServiceConfiguration { + public final static CommonVars LINKIS_AUTHOR_USER_TOKEN = CommonVars.apply("wds.linkis.client.api.service.author.user.token","WS-AUTH"); + public final static CommonVars LINKIS_ADMIN_USER = CommonVars.apply("wds.linkis.client.api.service.adminuser","ws"); + + public final static CommonVars LINKIS_CONNECTION_TIMEOUT = CommonVars.apply("wds.linkis.flow.connection.timeout",30000); + public final static CommonVars LINKIS_API_VERSION = CommonVars.apply("wds.linkis.server.version","v1"); + + +} diff --git a/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/dao/DataPipeDao.java b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/dao/DataPipeDao.java new file mode 100644 index 000000000..f1c065e99 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/dao/DataPipeDao.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.dao; + + + +public interface DataPipeDao { + + +} diff --git a/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/dao/mapper/DataPipeMapper.xml b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/dao/mapper/DataPipeMapper.xml new file mode 100644 index 000000000..a2c2837e2 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/dao/mapper/DataPipeMapper.xml @@ -0,0 +1,20 @@ + + + + + + diff --git a/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/exception/DataPipeExecuteException.java b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/exception/DataPipeExecuteException.java new file mode 100644 index 000000000..ea8445393 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/exception/DataPipeExecuteException.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class DataPipeExecuteException extends ErrorException { + + public DataPipeExecuteException(int errCode, String desc) { + super(errCode, desc); + } + + public DataPipeExecuteException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } + } diff --git a/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/execute/LinkisJobSubmit.java b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/execute/LinkisJobSubmit.java new file mode 100644 index 000000000..eb3dbb5d2 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/execute/LinkisJobSubmit.java @@ -0,0 +1,98 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.execute; + +import com.webank.wedatasphere.dss.datapipe.config.DataPipeServiceConfiguration; +import com.webank.wedatasphere.linkis.common.conf.Configuration; +import com.webank.wedatasphere.linkis.httpclient.dws.authentication.TokenAuthenticationStrategy; +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfig; +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfigBuilder; +import com.webank.wedatasphere.linkis.server.socket.controller.ServerEvent; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.UJESClientImpl; +import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; + +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.TimeUnit; + + +public class LinkisJobSubmit { + + public static UJESClient getClient(Map props) { + UJESClient client = getUJESClient( + Configuration.GATEWAY_URL().getValue(props), + DataPipeServiceConfiguration.LINKIS_ADMIN_USER.getValue(props), + DataPipeServiceConfiguration.LINKIS_AUTHOR_USER_TOKEN.getValue(props), + props); + + return client; + } + + public static UJESClient getUJESClient(String url, String user, String token, Map jobProps){ + return new UJESClientImpl(getClientConfig(url,user,token, jobProps)); + } + + public static DWSClientConfig getClientConfig(String url, String user, String token, Map jobProps){ + DWSClientConfig clientConfig = ((DWSClientConfigBuilder) (DWSClientConfigBuilder.newBuilder() + .addServerUrl(url) + .connectionTimeout(DataPipeServiceConfiguration.LINKIS_CONNECTION_TIMEOUT.getValue(jobProps)) + .discoveryEnabled(false).discoveryFrequency(1, TimeUnit.MINUTES) + .loadbalancerEnabled(true) + .maxConnectionSize(5) + .retryEnabled(false).readTimeout(DataPipeServiceConfiguration.LINKIS_CONNECTION_TIMEOUT.getValue(jobProps)) + .setAuthenticationStrategy(new TokenAuthenticationStrategy()) + .setAuthTokenKey(user).setAuthTokenValue(token))) + .setDWSVersion(DataPipeServiceConfiguration.LINKIS_API_VERSION.getValue(jobProps)).build(); + return clientConfig; + } + + /** + * @param operation + * @param client + * @return + */ + public static JobExecuteResult execute(ServerEvent operation, UJESClient client) { + Map dataMap = operation.getData(); + String runType = (String)dataMap.get("runType"); + String background = (String)dataMap.get("background"); + String executionCode = (String)dataMap.get("executionCode"); + String scriptPath = "default.scala"; + Map sourceMap = (Map)dataMap.get("source"); + scriptPath = sourceMap!=null && sourceMap.containsKey("scriptPath") ? sourceMap.get("scriptPath") : scriptPath; + + String executeApplicationName = (String)dataMap.get("executeApplicationName"); + String umUser = (String)dataMap.get("umUser"); + Map paramsMap = (Map)dataMap.get("params"); + + Map source = new HashMap<>(); + source.put("DSS-DataPipe",scriptPath); + JobExecuteAction.Builder builder = JobExecuteAction.builder().setCreator("IDE") + .addExecuteCode(executionCode) + .setEngineTypeStr(executeApplicationName) + .setRunTypeStr(runType) + .setUser(umUser) + .setParams(paramsMap) + .setVariableMap(new HashMap<>()) + .setSource(source); + JobExecuteAction jobAction = builder.build(); + JobExecuteResult res = client.execute(jobAction); + return res; + } + +} diff --git a/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/restful/DSSEntranceRestfulApi.java b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/restful/DSSEntranceRestfulApi.java new file mode 100644 index 000000000..bf5cb026b --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/java/com/webank/wedatasphere/dss/datapipe/restful/DSSEntranceRestfulApi.java @@ -0,0 +1,104 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.restful; + +import com.google.gson.Gson; +import com.google.gson.internal.LinkedTreeMap; +import com.webank.wedatasphere.dss.datapipe.entrance.background.BackGroundService; +import com.webank.wedatasphere.dss.datapipe.entrance.background.ExportBackGroundService; +import com.webank.wedatasphere.dss.datapipe.entrance.background.LoadBackGroundService; +import com.webank.wedatasphere.dss.datapipe.execute.LinkisJobSubmit; +import com.webank.wedatasphere.linkis.protocol.constants.TaskConstant; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import com.webank.wedatasphere.linkis.server.socket.controller.ServerEvent; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.Consumes; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.HashMap; +import java.util.Map; + + +@Path("/dss/datapipe") +@Component +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSEntranceRestfulApi { + private static final Logger logger = LoggerFactory.getLogger(DSSEntranceRestfulApi.class); + + @POST + @Path("/backgroundservice") + public Response backgroundservice(@Context HttpServletRequest req, Map json) { + Message message = null; + logger.info("Begin to get an execID"); + String backgroundType = (String) json.get("background"); + BackGroundService bgService = null; + if("export".equals(backgroundType)){ + bgService = new ExportBackGroundService(); + }else if("load".equals(backgroundType)){ + bgService = new LoadBackGroundService(); + }else{ + message = Message.error("export type is not exist："+backgroundType); + return Message.messageToResponse(message); + } + Gson gson = new Gson(); + Map executionCode = (Map) json.get("executionCode"); + executionCode = gson.fromJson(gson.toJson(executionCode),LinkedTreeMap.class); + json.put("executionCode",executionCode); + json.put(TaskConstant.UMUSER, SecurityFilter.getLoginUsername(req)); + ServerEvent serverEvent = new ServerEvent(); + serverEvent.setData(json); + serverEvent.setUser(SecurityFilter.getLoginUsername(req)); + ServerEvent operation = bgService.operation(serverEvent); + + + JobExecuteResult jobExecuteResult = toLinkisEntrance(operation); + message = Message.ok(); + message.setMethod("/api/dss/datapipe/backgroundservice"); + message.data("operation",operation); + message.data("execID", jobExecuteResult.getExecID()); + message.data("taskID", jobExecuteResult.getTaskID()); + logger.info("End to get an an execID: {}, taskID: {}", jobExecuteResult.getExecID(), jobExecuteResult.getTaskID()); + return Message.messageToResponse(message); + + } + + public JobExecuteResult toLinkisEntrance(ServerEvent operation){ + JobExecuteResult jobExecuteResult = null; + try{ + Map props = new HashMap<>(); + UJESClient client = LinkisJobSubmit.getClient(props); + jobExecuteResult = LinkisJobSubmit.execute(operation,client); + }catch (Exception e){ + logger.error("toLinkisEntranceError-",e); + throw e; + } + return jobExecuteResult; + + } +} diff --git a/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/DSSDatapipeServerApplication.scala b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/DSSDatapipeServerApplication.scala new file mode 100644 index 000000000..56b0553ed --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/DSSDatapipeServerApplication.scala @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe + +import com.webank.wedatasphere.dss.common.utils.DSSMainHelper +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} + + +object DSSDatapipeServerApplication extends Logging { + + val userName: String = System.getProperty("user.name") + val hostName: String = Utils.getComputerName + + def main(args: Array[String]): Unit = { + val serviceName = System.getProperty("serviceName")//ProjectConf.SERVICE_NAME.getValue + DSSMainHelper.formatPropertyFiles(serviceName) + val allArgs = args ++ DSSMainHelper.getExtraSpringOptions + System.setProperty("hostName", hostName) + System.setProperty("userName", userName) + info(s"Ready to start $serviceName with args: ${allArgs.toList}.") + println(s"Test Ready to start $serviceName with args: ${allArgs.toList}.") + DataWorkCloudApplication.main(allArgs) + } +} \ No newline at end of file diff --git a/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/AbstractBackGroundService.scala b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/AbstractBackGroundService.scala new file mode 100644 index 000000000..3e0b97dac --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/AbstractBackGroundService.scala @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.entrance.background + + +abstract class AbstractBackGroundService extends BackGroundService{ + +} diff --git a/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/BackGroundService.scala b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/BackGroundService.scala new file mode 100644 index 000000000..01af25e8b --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/BackGroundService.scala @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.entrance.background + +import com.webank.wedatasphere.linkis.server.socket.controller.ServerEvent + + +trait BackGroundService { + val serviceType:String + def operation(serverEvent: ServerEvent):ServerEvent +} diff --git a/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/BackGroundServiceUtils.scala b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/BackGroundServiceUtils.scala new file mode 100644 index 000000000..6fa27fd9c --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/BackGroundServiceUtils.scala @@ -0,0 +1,113 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.entrance.background + +import java.io.{InputStream, OutputStream} +import java.lang +import java.lang.reflect.Type +import java.text.SimpleDateFormat +import java.util.Date + +import com.fasterxml.jackson.databind.ObjectMapper +import com.google.gson.{GsonBuilder, JsonElement, JsonPrimitive, JsonSerializationContext, JsonSerializer} +import com.webank.wedatasphere.linkis.common.conf.CommonVars +import com.webank.wedatasphere.linkis.common.io.FsPath +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import com.webank.wedatasphere.linkis.storage.FSFactory +import com.webank.wedatasphere.linkis.storage.fs.FileSystem +import com.webank.wedatasphere.linkis.storage.utils.FileSystemUtils +import org.apache.commons.io.IOUtils +import org.apache.commons.lang.time.DateFormatUtils + +object BackGroundServiceUtils extends Logging { + + private val CODE_STORE_PREFIX = CommonVars("bdp.dataworkcloud.bgservice.store.prefix", "hdfs:///tmp/bdp-ide/") + private val CODE_STORE_SUFFIX = CommonVars("bdp.dataworkcloud.bgservice.store.suffix", "") + private val CHARSET = "utf-8" + private val CODE_SPLIT = ";" + private val LENGTH_SPLIT = "#" + + def storeExecutionCode(destination: String,user:String): String = { + if (destination.length < 60000) return null + val path: String = getCodeStorePath(user) + val fsPath: FsPath = new FsPath(path) + val fileSystem = FSFactory.getFsByProxyUser(fsPath, user).asInstanceOf[FileSystem] + fileSystem.init(null) + var os: OutputStream = null + var position = 0L + val codeBytes = destination.getBytes(CHARSET) + Utils.tryFinally { + path.intern() synchronized { + if (!fileSystem.exists(fsPath)) FileSystemUtils.createNewFile(fsPath, user, true) + os = fileSystem.write(fsPath, false) + position = fileSystem.get(path).getLength + IOUtils.write(codeBytes, os) + } + } { + if (fileSystem != null) fileSystem.close() + IOUtils.closeQuietly(os) + } + val length = codeBytes.length + path + CODE_SPLIT + position + LENGTH_SPLIT + length + } + + def exchangeExecutionCode(codePath: String): Unit = { + import scala.util.control.Breaks._ + val path = codePath.substring(0, codePath.lastIndexOf(CODE_SPLIT)) + val codeInfo = codePath.substring(codePath.lastIndexOf(CODE_SPLIT) + 1) + val infos: Array[String] = codeInfo.split(LENGTH_SPLIT) + val position = infos(0).toInt + var lengthLeft = infos(1).toInt + val tub = new Array[Byte](1024) + val executionCode: StringBuilder = new StringBuilder + val fsPath: FsPath = new FsPath(path) + val fileSystem = FSFactory.getFsByProxyUser(fsPath, System.getProperty("user.name")).asInstanceOf[FileSystem] + fileSystem.init(null) + var is: InputStream = null + if(!fileSystem.exists(fsPath)) return + Utils.tryFinally { + is = fileSystem.read(fsPath) + if (position > 0) is.skip(position) + breakable { + while (lengthLeft > 0) { + val readed = is.read(tub) + val useful = Math.min(readed, lengthLeft) + if (useful < 0) break() + lengthLeft -= useful + executionCode.append(new String(tub, 0, useful, CHARSET)) + } + } + } { + if (fileSystem != null) fileSystem.close() + IOUtils.closeQuietly(is) + } + executionCode.toString() + } + + private def getCodeStorePath(user: String): String = { + val date: String = DateFormatUtils.format(new Date, "yyyyMMdd") + s"${CODE_STORE_PREFIX.getValue}${user}${CODE_STORE_SUFFIX.getValue}/executionCode/${date}/_bgservice" + } + + implicit val gson = new GsonBuilder().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").serializeNulls + .registerTypeAdapter(classOf[java.lang.Double], new JsonSerializer[java.lang.Double] { + override def serialize(t: lang.Double, `type`: Type, jsonSerializationContext: JsonSerializationContext): JsonElement = + if(t == t.longValue()) new JsonPrimitive(t.longValue()) else new JsonPrimitive(t) + }).create + + implicit val jacksonJson = new ObjectMapper().setDateFormat(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ")) +} diff --git a/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/ExportBackGroundService.scala b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/ExportBackGroundService.scala new file mode 100644 index 000000000..a51350067 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/ExportBackGroundService.scala @@ -0,0 +1,101 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.entrance.background + +import java.util + +import com.google.gson.internal.LinkedTreeMap +import com.google.gson.{JsonObject, JsonParser} +import com.webank.wedatasphere.linkis.common.utils.Logging +import com.webank.wedatasphere.linkis.server._ +import com.webank.wedatasphere.linkis.server.socket.controller.ServerEvent + + +class ExportBackGroundService extends AbstractBackGroundService with Logging{ + override val serviceType: String = "export" + + override def operation(serverEvent: ServerEvent): ServerEvent = { + val params = serverEvent.getData.map { case (k, v) => k -> v.asInstanceOf[Any] } + //val executionCode = params.get("executionCode").get + val ec = params.get("executionCode").get.asInstanceOf[LinkedTreeMap[String,LinkedTreeMap[String,String]]] + if(ec.get("destination")!=null && ec.get("destination").get("fieldDelimiter") != null){ + info(s"---${ec.get("destination").get("fieldDelimiter")}---") + ec.get("destination").get("fieldDelimiter") match { + case "\\t" =>ec.get("destination").put("fieldDelimiter","\t") + case _ =>info("---other fieldDelimiter---") + } + } + val executionCode = BackGroundServiceUtils.gson.toJson(params.get("executionCode").get) + // TODO: Head may be removed + var newExecutionCode = "" + val jsonParser = new JsonParser() + val jsonCode = jsonParser.parse(executionCode.asInstanceOf[String]).asInstanceOf[JsonObject] + val destination = "val destination = \"\"\"" + jsonCode.get("destination").toString + "\"\"\"\n" + val dataInfo = jsonCode.get("dataInfo").toString + var newDataInfo = "val dataInfo = \"\"\"" + val storePath = BackGroundServiceUtils.storeExecutionCode(dataInfo,serverEvent.getUser) + if(storePath == null) newDataInfo += dataInfo + "\"\"\"\n" else newDataInfo += storePath + "\"\"\"\n" + newExecutionCode += destination + newExecutionCode += newDataInfo + if(storePath == null) + newExecutionCode += "com.webank.wedatasphere.linkis.engineplugin.spark.imexport.ExportData.exportData(spark,dataInfo,destination)" + else + newExecutionCode += "com.webank.wedatasphere.linkis.engineplugin.spark.imexport.ExportData.exportDataByFile(spark,dataInfo,destination)" + params.put("executionCode", newExecutionCode) + print(newExecutionCode) + val map = new util.HashMap[String, Object]() + params.foreach(f => map.put(f._1, f._2.asInstanceOf[Object])) + serverEvent.setData(map) + serverEvent + } + + def splitDataInfo(dataInfo:String):util.ArrayList[String] = { + val length = 6000 + val list = new util.ArrayList[String]() + var size = dataInfo.length /length + if(dataInfo.length % length != 0) size += 1 + for(i <- 0 to size-1){ + list.add(subString(dataInfo,i * length,(i +1) * length)) + } + list + } + + private def subString(str:String,begin:Int,end:Int):String = { + if(begin > str.length) return null + if(end > str.length) return str.substring(begin,str.length) + str.substring(begin,end) + } + + +} + +object A{ + def main(args: Array[String]): Unit = { + val builder: StringBuilder = new StringBuilder + for(i <- 1 to 200){ + if(i == 1) builder.append(2) else builder.append(1) + + } + var newDataInfo = "val dataInfo = \"\"\"" + val service: ExportBackGroundService = new ExportBackGroundService() + val splitDataInfos: util.ArrayList[String] =service.splitDataInfo(builder.toString()) + val splitString = "\"\"\"" + "+" + "\"\"\"" + val compaction = splitDataInfos.foldLeft("")((l,r) => l + splitString + r ) + newDataInfo += compaction.substring(splitString.length,compaction.length) + "\"\"\"\n" + print(newDataInfo) + } +} diff --git a/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/LoadBackGroundService.scala b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/LoadBackGroundService.scala new file mode 100644 index 000000000..fa2111355 --- /dev/null +++ b/dss-apps/dss-datapipe-server/src/main/scala/com/webank/wedatasphere/dss/datapipe/entrance/background/LoadBackGroundService.scala @@ -0,0 +1,74 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.datapipe.entrance.background + +import java.util + +import com.google.gson.{JsonObject, JsonParser} +import com.webank.wedatasphere.linkis.server._ +import com.webank.wedatasphere.linkis.server.socket.controller.ServerEvent + + +class LoadBackGroundService extends AbstractBackGroundService { + + override val serviceType: String = "load" + + override def operation(serverEvent: ServerEvent): ServerEvent = { + val params = serverEvent.getData.map { case (k, v) => k -> v.asInstanceOf[Any] } + val executionCode = BackGroundServiceUtils.gson.toJson(params.get("executionCode").get) + // TODO: Head may be removed + var newExecutionCode = "" + val jsonParser = new JsonParser() + val jsonCode = jsonParser.parse(executionCode.asInstanceOf[String]).asInstanceOf[JsonObject] + val source = "val source = \"\"\"" + jsonCode.get("source").toString + "\"\"\"\n" + val destination = jsonCode.get("destination").toString + var newDestination = "val destination = \"\"\"" + val storePath = BackGroundServiceUtils.storeExecutionCode(destination,serverEvent.getUser) + if(storePath == null) newDestination +=destination +"\"\"\"\n" else newDestination +=storePath + "\"\"\"\n" + newExecutionCode += source + newExecutionCode += newDestination + if(storePath == null){ + newExecutionCode += "com.webank.wedatasphere.linkis.engineplugin.spark.imexport.LoadData.loadDataToTable(spark,source,destination)" + }else{ + newExecutionCode += "com.webank.wedatasphere.linkis.engineplugin.spark.imexport.LoadData.loadDataToTableByFile(spark,destination,source)" + } + + params.put("executionCode", newExecutionCode) + print(newExecutionCode) + val map = new util.HashMap[String, Object]() + params.foreach(f => map.put(f._1, f._2.asInstanceOf[Object])) + serverEvent.setData(map) + serverEvent + } + + def splitDestination(destination:String):util.ArrayList[String] = { + val length = 6000 + val list = new util.ArrayList[String]() + var size = destination.length /length + if(destination.length % length != 0) size += 1 + for(i <- 0 to size-1){ + list.add(subString(destination,i * length,(i +1) * length)) + } + list + } + + private def subString(str:String,begin:Int,end:Int):String = { + if(begin > str.length) return null + if(end > str.length) return str.substring(begin,str.length) + str.substring(begin,end) + } +} diff --git a/dss-azkaban-scheduler-appjoint/pom.xml b/dss-azkaban-scheduler-appjoint/pom.xml deleted file mode 100644 index 4f67a636b..000000000 --- a/dss-azkaban-scheduler-appjoint/pom.xml +++ /dev/null @@ -1,105 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-azkaban-scheduler-appjoint - - - - - - - - - com.webank.wedatasphere.dss - dss-scheduler-appjoint-core - ${dss.version} - provided - true - - - org.apache.httpcomponents - httpmime - 4.5.4 - - - com.webank.wedatasphere.dss - dss-application - ${dss.version} - provided - true - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - dss-azkaban-appjoint - false - false - - src/main/assembly/distribution.xml - - - - - - - \ No newline at end of file diff --git a/dss-azkaban-scheduler-appjoint/src/main/assembly/distribution.xml b/dss-azkaban-scheduler-appjoint/src/main/assembly/distribution.xml deleted file mode 100644 index 289856124..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/assembly/distribution.xml +++ /dev/null @@ -1,136 +0,0 @@ - - - - dss-azkaban-appjoint - - zip - - true - schedulis - - - - - - lib - true - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ${basedir}/src/main/resources - - appjoint.properties - - 0777 - / - unix - - - - ${basedir}/src/main/resources - - log4j.properties - log4j2.xml - - 0777 - conf - unix - - - - - - diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/AzkabanSchedulerAppJoint.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/AzkabanSchedulerAppJoint.java deleted file mode 100644 index 8f6e4bb8a..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/AzkabanSchedulerAppJoint.java +++ /dev/null @@ -1,85 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban; - -import com.webank.wedatasphere.dss.appjoint.scheduler.SchedulerAppJoint; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.conf.AzkabanConf; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.hooks.LinkisAzkabanProjectPublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.parser.AzkabanProjectParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.tuning.AzkabanProjectTuning; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.ProjectPublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.ProjectParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.service.SchedulerProjectService; -import com.webank.wedatasphere.dss.appjoint.scheduler.service.SchedulerSecurityService; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.service.AzkabanProjectService; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.service.AzkabanSecurityService; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.ProjectTuning; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.util.StringUtils; -import java.io.IOException; -import java.util.Map; - -/** - * Created by cooperyang on 2019/9/16. - */ - -public final class AzkabanSchedulerAppJoint extends AppJointUrlImpl implements SchedulerAppJoint { - - private static final Logger LOGGER = LoggerFactory.getLogger(AzkabanSchedulerAppJoint.class); - - private SchedulerSecurityService securityService; - private SchedulerProjectService projectService; - private ApplicationService applicationService; - - @Override - public String getAppJointName() { - return "schedulis"; - } - - - @Override - public void init(String basicUrl, Map params) { - LOGGER.info("read schedulerAppJoint url from db{}",basicUrl); - if(StringUtils.isEmpty(basicUrl)){ - basicUrl = AzkabanConf.AZKABAN_BASE_URL.getValue(); - LOGGER.warn("basic url in db is empty,read it from conf{}",basicUrl); - } - securityService = new AzkabanSecurityService(); - securityService.setBaseUrl(basicUrl); - - projectService = new AzkabanProjectService(); - projectService.setBaseUrl(basicUrl); - } - - @Override - public SchedulerSecurityService getSecurityService() { - return this.securityService; - } - - @Override - public ProjectParser getProjectParser() { - return new AzkabanProjectParser(); - } - - @Override - public ProjectTuning getProjectTuning() { - return new AzkabanProjectTuning(); - } - - @Override - public ProjectPublishHook[] getProjectPublishHooks() { - ProjectPublishHook[] projectPublishHooks = {new LinkisAzkabanProjectPublishHook()}; - return projectPublishHooks; - } - - @Override - public SchedulerProjectService getProjectService() { - return this.projectService; - } - - @Override - public void close() throws IOException { - - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/conf/AzkabanConf.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/conf/AzkabanConf.java deleted file mode 100644 index 49ea9948a..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/conf/AzkabanConf.java +++ /dev/null @@ -1,14 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.conf; - - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; - -/** - * Created by cooperyang on 2019/9/16. - */ -public class AzkabanConf { - - public static final CommonVars AZKABAN_BASE_URL = CommonVars.apply("wds.dss.appjoint.scheduler.azkaban.address", ""); - public static final CommonVars DEFAULT_STORE_PATH = CommonVars.apply("wds.dss.appjoint.scheduler.project.store.dir", "/appcom/tmp/wds/dss"); - public static final CommonVars AZKABAN_LOGIN_PWD = CommonVars.apply("wds.dss.appjoint.scheduler.azkaban.login.passwd", "password"); -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/constant/AzkabanConstant.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/constant/AzkabanConstant.java deleted file mode 100644 index eda5820b5..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/constant/AzkabanConstant.java +++ /dev/null @@ -1,17 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.constant; - -/** - * Created by cooperyang on 2019/9/27. - */ -public class AzkabanConstant { - public final static String LINKIS_FLOW_VARIABLE_KEY = "flow.variable."; - public final static String AZKABAN_JOB_SUFFIX = ".job"; - public final static String AZKABAN_PROPERTIES_SUFFIX = ".properties"; - public final static String LINKIS_JOB_RESOURCES_KEY = "resources="; - public final static String ZAKABAN_DEPENDENCIES_KEY = "dependencies"; - public final static String TOKEN_FILE_NAME = "token.properties"; - public final static String JOB_TYPE = "type"; - public final static String LINKIS_TYPE = "linkistype"; - public final static String JOB_COMMAND = "command"; - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerFlow.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerFlow.java deleted file mode 100644 index a8a6aae1d..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerFlow.java +++ /dev/null @@ -1,17 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; - -public class AzkabanSchedulerFlow extends SchedulerFlow { - - private String storePath; - - public String getStorePath() { - return storePath; - } - - public void setStorePath(String storePath) { - this.storePath = storePath; - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerNode.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerNode.java deleted file mode 100644 index d4b3861d0..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerNode.java +++ /dev/null @@ -1,22 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerNode; -import com.webank.wedatasphere.linkis.common.exception.ErrorException; - -/** - * Created by cooperyang on 2019/9/15 - */ -public class AzkabanSchedulerNode extends AbstractSchedulerNode { - - private String storePath; - - public String getStorePath() { - return storePath; - } - - public void setStorePath(String storePath) { - this.storePath = storePath; - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerProject.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerProject.java deleted file mode 100644 index 4e28e2655..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/AzkabanSchedulerProject.java +++ /dev/null @@ -1,21 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerProject; - -/** - * Created by cooperyang on 2019/9/19 - */ -public class AzkabanSchedulerProject extends AbstractSchedulerProject { - - private String storePath; - public static final String DATE_FORMAT = "yyyy-MM-dd"; - - public String getStorePath() { - return storePath; - } - - public void setStorePath(String storePath) { - this.storePath = storePath; - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanReadNode.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanReadNode.java deleted file mode 100644 index ccee05d43..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanReadNode.java +++ /dev/null @@ -1,37 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob.LinkisJobConverter; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; - -public class LinkisAzkabanReadNode extends LinkisAzkabanSchedulerNode implements ReadNode { - - private SchedulerNode schedulerNode; - private String[] nodeIds; - - @Override - public String[] getShareNodeIds() { - return this.nodeIds; - } - - @Override - public void setShareNodeIds(String[] nodeIds) { - this.nodeIds = nodeIds; - } - - @Override - public SchedulerNode getSchedulerNode() { - return schedulerNode; - } - - @Override - public void setSchedulerNode(SchedulerNode schedulerNode) { - this.schedulerNode = schedulerNode; - } - - @Override - public String toJobString(LinkisJobConverter converter) { - return converter.conversion(this); - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanSchedulerNode.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanSchedulerNode.java deleted file mode 100644 index 750f4d1e1..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanSchedulerNode.java +++ /dev/null @@ -1,17 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob.LinkisJobConverter; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Created by cooperyang on 2019/9/29. - */ -public class LinkisAzkabanSchedulerNode extends AzkabanSchedulerNode { - - private final Logger logger = LoggerFactory.getLogger(this.getClass()); - - public String toJobString(LinkisJobConverter converter) { - return converter.conversion(this); - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanShareNode.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanShareNode.java deleted file mode 100644 index 44ac7d26b..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/entity/LinkisAzkabanShareNode.java +++ /dev/null @@ -1,35 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob.LinkisJobConverter; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; - -public class LinkisAzkabanShareNode extends LinkisAzkabanSchedulerNode implements ShareNode { - private SchedulerNode schedulerNode; - private int shareTimes; - - @Override - public int getShareTimes() { - return shareTimes; - } - - @Override - public void setShareTimes(int num) { - this.shareTimes = num; - } - - @Override - public SchedulerNode getSchedulerNode() { - return schedulerNode; - } - - @Override - public void setSchedulerNode(SchedulerNode node) { - this.schedulerNode = node; - } - - @Override - public String toJobString(LinkisJobConverter converter) { - return converter.conversion(this); - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanFlowPublishHook.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanFlowPublishHook.java deleted file mode 100644 index 73cec2164..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanFlowPublishHook.java +++ /dev/null @@ -1,113 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.hooks; - - -import com.google.gson.Gson; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.AbstractFlowPublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.NodePublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.constant.AzkabanConstant; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerFlow; -import com.webank.wedatasphere.dss.common.entity.Resource; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.apache.commons.io.FileUtils; -import org.apache.commons.io.IOUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import java.io.File; -import java.io.FileOutputStream; -import java.util.List; -import java.util.Map; - -/** - * Created by allenlliu on 2019/9/20. - */ - -public class LinkisAzkabanFlowPublishHook extends AbstractFlowPublishHook { - - private static final Logger LOGGER = LoggerFactory.getLogger(LinkisAzkabanFlowPublishHook.class); - - public LinkisAzkabanFlowPublishHook(){ - NodePublishHook[] nodePublishHooks = {new LinkisAzkabanNodePublishHook()}; - setNodeHooks(nodePublishHooks); - } - - @Override - public void prePublish(SchedulerFlow flow) throws DSSErrorException { - // 处理资源生成文件等等 - writeFlowResourcesToLocal(flow); - writeFlowPropertiesToLocal(flow); - super.prePublish(flow); - } - - private void writeFlowResourcesToLocal(SchedulerFlow flow) throws DSSErrorException { - List flowResources = flow.getFlowResources(); - FileOutputStream os = null; - try { - String storePath = ((AzkabanSchedulerFlow)flow).getStorePath(); - File flowDir = new File(storePath); - FileUtils.forceMkdir(flowDir); - if(flowResources == null || flowResources.isEmpty()) {return;} - String projectStorePath = getProjectStorePath(storePath); - String flowResourceStringPrefix = getFlowResourceStringPrefix(projectStorePath,storePath); - String flowtResourceString = flowResourceStringPrefix + new Gson().toJson(flowResources) + "\n"; - File projectResourcesFile = new File(projectStorePath, "project.properties"); - os = FileUtils.openOutputStream(projectResourcesFile,true); - os.write(flowtResourceString.getBytes()); - }catch (Exception e){ - LOGGER.error("write FlowResources to local failed,reason:",e); - throw new DSSErrorException(90006,e.getMessage()); - }finally { - IOUtils.closeQuietly(os); - } - } - - private String getFlowResourceStringPrefix(String projectStorePath, String storePath) { - // TODO: 2019/9/30 需要做用户工作流命名为subFlows的情况的判断 - String substring = storePath.substring(projectStorePath.length() + 1); - String prefix = substring.replaceAll("\\" + File.separator +"subFlows" + "\\" + File.separator,"."); - return "flow." + prefix + "_.resources="; - } - - private String getProjectStorePath(String storePath) { - int indexOf = storePath.indexOf("subFlows"); - if(indexOf != -1){ - storePath = storePath.substring(0, indexOf-1); - } - return storePath.substring(0, storePath.lastIndexOf(File.separator)); - } - - private void writeFlowPropertiesToLocal(SchedulerFlow flow)throws DSSErrorException { - List> flowProperties = flow.getFlowProperties(); - if(flowProperties == null || flowProperties.isEmpty()) {return;} - FileOutputStream os = null; - try { - String storePath = ((AzkabanSchedulerFlow)flow).getStorePath(); - File flowPrpsFile = new File(storePath,flow.getName() + AzkabanConstant.AZKABAN_PROPERTIES_SUFFIX); - flowPrpsFile.createNewFile(); - os = FileUtils.openOutputStream(flowPrpsFile,true); - StringBuilder stringBuilder = new StringBuilder(); - flowProperties.forEach(p ->p.forEach((k,v)->{ - stringBuilder.append(AzkabanConstant.LINKIS_FLOW_VARIABLE_KEY + k + "=" + v + "\n"); - })); - os.write(stringBuilder.toString().getBytes()); - }catch (Exception e){ - LOGGER.error("write flowProperties to local faailed,reason:",e); - throw new DSSErrorException(90007,e.getMessage()); - }finally { - IOUtils.closeQuietly(os); - } - } - - @Override - public void postPublish(SchedulerFlow flow) { - super.postPublish(flow); - } - - - @Override - public void setNodeHooks(NodePublishHook[] nodePublishHooks) { - super.setNodeHooks(nodePublishHooks); - } - - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanNodePublishHook.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanNodePublishHook.java deleted file mode 100644 index 702275f40..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanNodePublishHook.java +++ /dev/null @@ -1,79 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.hooks; - -import com.google.gson.Gson; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanSchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.AbstractNodePublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.constant.AzkabanConstant; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerNode; -import com.webank.wedatasphere.dss.common.entity.Resource; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob.LinkisJobConverter; -import org.apache.commons.io.FileUtils; -import org.apache.commons.io.IOUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import java.io.File; -import java.io.FileOutputStream; -import java.util.List; - - -public class LinkisAzkabanNodePublishHook extends AbstractNodePublishHook { - - private static final Logger LOGGER = LoggerFactory.getLogger(LinkisAzkabanNodePublishHook.class); - private LinkisJobConverter linkisJobConverter; - - public LinkisAzkabanNodePublishHook(){ - this.linkisJobConverter = new LinkisJobConverter(); - } - - @Override - public void prePublish(SchedulerNode schedulerNode) throws DSSErrorException { - writeNodeTojobLocal(schedulerNode); - writeNodeResourcesToLocal(schedulerNode); - } - - private void writeNodeTojobLocal(SchedulerNode schedulerNode) throws DSSErrorException { - LinkisAzkabanSchedulerNode azkabanSchedulerNode = (LinkisAzkabanSchedulerNode) schedulerNode; - FileOutputStream os = null; - try { - String storePath = azkabanSchedulerNode.getStorePath(); - File jobDirFile = new File(storePath); - FileUtils.forceMkdir(jobDirFile); - File jobFile = new File(storePath,schedulerNode.getName() + AzkabanConstant.AZKABAN_JOB_SUFFIX); - jobFile.createNewFile(); - String nodeString = azkabanSchedulerNode.toJobString(linkisJobConverter); - os = FileUtils.openOutputStream(jobFile,true); - os.write(nodeString.getBytes()); - }catch (Exception e){ - LOGGER.error("write AppJointNode to jobLocal failed,reason:",e); - throw new DSSErrorException(90017,e.getMessage()); - }finally { - IOUtils.closeQuietly(os); - } - } - - @Override - public void postPublish(SchedulerNode schedulerNode) { - - } - - private void writeNodeResourcesToLocal(SchedulerNode schedulerNode) throws DSSErrorException { - List nodeResources = schedulerNode.getDssNode().getResources(); - if(nodeResources == null || nodeResources.isEmpty()) {return;} - FileOutputStream os = null; - try { - String storePath = ((AzkabanSchedulerNode)schedulerNode).getStorePath(); - File jobFile = new File(storePath,schedulerNode.getName() + AzkabanConstant.AZKABAN_JOB_SUFFIX); - String nodeResourceString = AzkabanConstant.LINKIS_JOB_RESOURCES_KEY + new Gson().toJson(nodeResources); - os = FileUtils.openOutputStream(jobFile,true); - os.write(nodeResourceString.getBytes()); - }catch (Exception e){ - LOGGER.error("write nodeResources to local failed,reason:",e); - throw new DSSErrorException(90018,e.getMessage()); - }finally { - IOUtils.closeQuietly(os); - } - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanProjectPublishHook.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanProjectPublishHook.java deleted file mode 100644 index 40effab61..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/hooks/LinkisAzkabanProjectPublishHook.java +++ /dev/null @@ -1,127 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.hooks; - -import com.google.gson.Gson; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.util.AzkabanUtilsScala; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProjectVersionForFlows; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.AbstractProjectPublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.FlowPublishHook; -import com.webank.wedatasphere.dss.common.entity.Resource; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.apache.commons.io.FileUtils; -import org.apache.commons.io.IOUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import java.io.File; -import java.io.FileOutputStream; -import java.util.ArrayList; -import java.util.List; - -/** - * Created by cooperyang on 2019/9/18 - */ - -public class LinkisAzkabanProjectPublishHook extends AbstractProjectPublishHook { - - private static final Logger LOGGER = LoggerFactory.getLogger( LinkisAzkabanProjectPublishHook.class); - public LinkisAzkabanProjectPublishHook(){ - FlowPublishHook[] flowPublishHooks = {new LinkisAzkabanFlowPublishHook()}; - setFlowPublishHooks(flowPublishHooks); - } - - @Override - public void setFlowPublishHooks(FlowPublishHook[] flowPublishHooks) { - super.setFlowPublishHooks(flowPublishHooks); - } - - @Override - public void prePublish(SchedulerProject project) throws DSSErrorException { - //1.检查重复的节点名 - AzkabanSchedulerProject publishProject = (AzkabanSchedulerProject)project; - SchedulerProjectVersionForFlows schedulerProjectVersionForFlows = publishProject.getProjectVersions().get(0); - List allSchedulerFlows = schedulerProjectVersionForFlows.getFlows(); - List repeatNode = AzkabanUtilsScala.getRepeatNodeName(getAllNodeName(allSchedulerFlows)); - if(repeatNode.size()>0){ - throw new DSSErrorException(80001,"重复的节点名称："+repeatNode.toString()); - } - //删除可能未处理的zip包和文件夹 - removeProjectStoreDirAndzip(publishProject); - // 2.处理资源 - writeProjectResourcesToLocal(publishProject); - //3.调用flowPublishHooks - super.prePublish(project); - } - - private void removeProjectStoreDirAndzip(AzkabanSchedulerProject publishProject) throws DSSErrorException { - String storePath = publishProject.getStorePath(); - File projectDir = new File(storePath); - try { - if(projectDir.exists()) { - LOGGER.info("exist project dir{} before publish ,now remove it",storePath); - FileUtils.deleteDirectory(projectDir); - } - String projectZip = projectDir.getParent() + File.separator + publishProject.getName() + ".zip"; - File zipFile = new File(projectZip); - if(zipFile.exists()){ - LOGGER.info("exist project zip{} before publish ,now remove it",projectZip); - zipFile.delete(); - } - }catch (Exception e){ - LOGGER.error("delete project dir or zip failed,reaseon:",e); - throw new DSSErrorException(90020,e.getMessage()); - } - } - - private void writeProjectResourcesToLocal(AzkabanSchedulerProject publishProject)throws DSSErrorException { - List resources = publishProject.getDssProject().getProjectResources(); - FileOutputStream os = null; - try { - String storePath = publishProject.getStorePath(); - File projectDir = new File(storePath); - FileUtils.forceMkdir(projectDir); - File projectResourcesFile = new File(storePath, "project.properties"); - projectResourcesFile.createNewFile(); - if(resources == null || resources.isEmpty()) {return;} - String projectResourceStringPrefix = "project." + publishProject.getName() + ".resources="; - String projectResourceString = projectResourceStringPrefix + new Gson().toJson(resources) + "\n"; - os = FileUtils.openOutputStream(projectResourcesFile); - os.write(projectResourceString.getBytes()); - }catch (Exception e){ - LOGGER.error("write projectResources to local failed,reason:",e); - throw new DSSErrorException(90015,e.getMessage()); - }finally { - IOUtils.closeQuietly(os); - } - } - - @Override - public void postPublish(SchedulerProject project) throws DSSErrorException { - // TODO: 2019/9/26 1.删除zip包和文件夹 - AzkabanSchedulerProject publishProject = (AzkabanSchedulerProject)project; - String storePath = publishProject.getStorePath(); - try { - String projectName = publishProject.getName(); - File file = new File(storePath); - new File(file.getParent() + File.separator + projectName + ".zip" ).delete(); - }catch (Exception e){ - LOGGER.error("删除发布后的zip包失败",e); - throw new DSSErrorException(90001,"发布后删除工程目录失败"); - } - // TODO: 2019/9/26 flowPublishHooks ... - } - - /** - * 直接从allFlows 中获取所有节点的名字，免去一次递归 - * @param allSchedulerFlows - * @return - */ - public List getAllNodeName(List allSchedulerFlows) { - List nodeNames = new ArrayList<>(); - allSchedulerFlows.forEach(flow ->flow.getSchedulerNodes().forEach(node ->nodeNames.add(node.getName()))); - return nodeNames; - } - - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/AzkabanSubFlowJobTuning.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/AzkabanSubFlowJobTuning.java deleted file mode 100644 index 4b31aa9dd..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/AzkabanSubFlowJobTuning.java +++ /dev/null @@ -1,23 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob; - - - -/** - * Created by cooperyang on 2019/11/1. - */ - -public class AzkabanSubFlowJobTuning implements LinkisJobTuning { - - @Override - public LinkisJob tuningJob(LinkisJob job) { - job.setType("flow"); - job.setLinkistype(null); - job.getConf().put("flow.name",job.getName() + "_"); - return job; - } - - @Override - public boolean ifJobCantuning(String nodeType) { - return "workflow.subflow".equals(nodeType); - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJob.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJob.java deleted file mode 100644 index fb19c0577..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJob.java +++ /dev/null @@ -1,74 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob; - -import java.util.HashMap; -import java.util.Map; - -/** - * Created by cooperyang on 2019/11/1. - */ -public class LinkisJob { - private String name; - private String type; - private String linkistype; - private String proxyUser; - private String dependencies; - private Map conf; - private String command; - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getType() { - return type; - } - - public void setType(String type) { - this.type = type; - } - - public String getLinkistype() { - return linkistype; - } - - public void setLinkistype(String linkistype) { - this.linkistype = linkistype; - } - - public String getProxyUser() { - return proxyUser; - } - - public void setProxyUser(String proxyUser) { - this.proxyUser = proxyUser; - } - - public String getDependencies() { - return dependencies; - } - - public void setDependencies(String dependencies) { - this.dependencies = dependencies; - } - - public Map getConf() { - return conf; - } - - public void setConf(Map conf) { - this.conf = conf; - } - - public String getCommand() { - return command; - } - - public void setCommand(String command) { - this.command = command; - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJobConverter.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJobConverter.java deleted file mode 100644 index ef6f3f3b1..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJobConverter.java +++ /dev/null @@ -1,112 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob; - -import com.google.gson.Gson; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.constant.AzkabanConstant; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanReadNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanSchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanShareNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.constant.SchedulerAppJointConstant; -import org.apache.commons.lang.StringUtils; -import java.util.Arrays; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -/** - * Created by cooperyang on 2019/11/1. - */ - -public class LinkisJobConverter { - - public LinkisJobConverter(){ - LinkisJobTuning[] linkisJobTunings = {new AzkabanSubFlowJobTuning()}; - this.linkisJobTunings = linkisJobTunings; - } - private LinkisJobTuning[] linkisJobTunings; - - public String conversion(LinkisAzkabanSchedulerNode schedulerNode){ - return baseConversion(schedulerNode); - } - - public String conversion(LinkisAzkabanReadNode schedulerNode){ - String tmpConversion = baseConversion(schedulerNode); - String[] shareNodeArray = schedulerNode.getShareNodeIds(); - String readNodePrefix = "read.nodes = "; - String shareNodeIds = StringUtils.join(shareNodeArray, ","); - return tmpConversion + readNodePrefix + shareNodeIds + "\n"; - } - - public String conversion(LinkisAzkabanShareNode schedulerNode){ - String tmpConversion = baseConversion(schedulerNode); - String shareNodePrefix = "share.num = "; - String shareNum = Integer.toString((schedulerNode).getShareTimes()); - return tmpConversion + shareNodePrefix + shareNum + "\n"; - } - - private String baseConversion(LinkisAzkabanSchedulerNode schedulerNode){ - LinkisJob job = new LinkisJob(); - job.setConf(new HashMap()); - job.setName(schedulerNode.getName()); - convertHead(schedulerNode,job); - convertDependencies(schedulerNode,job); - convertProxyUser(schedulerNode,job); - convertConfiguration(schedulerNode,job); - convertJobCommand(schedulerNode,job); - Arrays.stream(linkisJobTunings).forEach(t ->{ - if(t.ifJobCantuning(schedulerNode.getNodeType())) t.tuningJob(job); - }); - return convertJobtoString(job); - } - - private String convertJobtoString(LinkisJob job){ - HashMap map = new HashMap<>(); - map.put(AzkabanConstant.JOB_TYPE,job.getType()); - map.put(AzkabanConstant.LINKIS_TYPE,job.getLinkistype()); - map.put(AzkabanConstant.ZAKABAN_DEPENDENCIES_KEY,job.getDependencies()); - map.put(SchedulerAppJointConstant.PROXY_USER,job.getProxyUser()); - map.put(AzkabanConstant.JOB_COMMAND,job.getCommand()); - map.putAll(job.getConf()); - StringBuilder stringBuilder = new StringBuilder(); - map.forEach((k,v)->{ - if(v != null) stringBuilder.append(k).append("=").append(v).append("\n"); - }); - return stringBuilder.toString(); - } - - private void convertHead(LinkisAzkabanSchedulerNode schedulerNode,LinkisJob job){ - job.setType("linkis"); - job.setLinkistype(schedulerNode.getNodeType()); - } - - private void convertDependencies(LinkisAzkabanSchedulerNode schedulerNode,LinkisJob job){ - List dependencys = schedulerNode.getDssNode().getDependencys(); - if(dependencys != null && !dependencys.isEmpty()) { - StringBuilder dependencies = new StringBuilder(); - dependencys.forEach(d ->dependencies.append(d + ",")); - job.setDependencies(dependencies.substring(0,dependencies.length()-1)); - } - } - - private void convertProxyUser(LinkisAzkabanSchedulerNode schedulerNode,LinkisJob job){ - String userProxy = schedulerNode.getDssNode().getUserProxy(); - if(!StringUtils.isEmpty(userProxy)) job.setProxyUser(userProxy); - } - - private void convertConfiguration(LinkisAzkabanSchedulerNode schedulerNode,LinkisJob job){ - Map params = schedulerNode.getDssNode().getParams(); - if (params != null && !params.isEmpty()) { - Object configuration = params.get("configuration"); - String confprefix = "node.conf."; - ((Map>)configuration).forEach((k,v)->v.forEach((k2,v2)->job.getConf().put(confprefix + k + "." + k2,v2.toString()))); - } - - } - - private void convertJobCommand(LinkisAzkabanSchedulerNode schedulerNode,LinkisJob job){ - Map jobContent = schedulerNode.getDssNode().getJobContent(); - if(jobContent != null) { - jobContent.remove("jobParams"); - job.setCommand(new Gson().toJson(jobContent)); - } - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJobTuning.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJobTuning.java deleted file mode 100644 index 0884c1b5e..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/linkisjob/LinkisJobTuning.java +++ /dev/null @@ -1,11 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.linkisjob; - -/** - * Created by cooperyang on 2019/11/1. - */ -public interface LinkisJobTuning { - - LinkisJob tuningJob(LinkisJob job); - - boolean ifJobCantuning(String nodeType); -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanFlowParser.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanFlowParser.java deleted file mode 100644 index 0fe735083..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanFlowParser.java +++ /dev/null @@ -1,38 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractFlowParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.NodeParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSJSONFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import java.util.ArrayList; - - -public class AzkabanFlowParser extends AbstractFlowParser { - - public AzkabanFlowParser(){ - ArrayList list = new ArrayList<>(); - list.add(new LinkisAzkabanNodeParser()); - list.add(new LinkisAzkabanSendEmailNodeParser()); - NodeParser[] nodeParsers =new NodeParser[list.size()]; - list.toArray(nodeParsers); - setNodeParsers(nodeParsers); - } - - @Override - protected SchedulerFlow createSchedulerFlow() { - return new AzkabanSchedulerFlow(); - } - - @Override - public void setNodeParsers(NodeParser[] nodeParsers) { - super.setNodeParsers(nodeParsers); - } - - @Override - public Boolean ifFlowCanParse(DSSJSONFlow flow) { - return true; - } - - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanNodeParser.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanNodeParser.java deleted file mode 100644 index 156f14ba5..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanNodeParser.java +++ /dev/null @@ -1,17 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.parser; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerEdge; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractNodeParser; - -import java.util.List; - -public abstract class AzkabanNodeParser extends AbstractNodeParser { - - protected Boolean generateDependencies(SchedulerNode node, List flowEdges){return null;}; - - protected Boolean assertNodeType(){return null;}; - - protected String getNodeResourcesStr(){return null;}; -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanProjectParser.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanProjectParser.java deleted file mode 100644 index 3895d7c68..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/AzkabanProjectParser.java +++ /dev/null @@ -1,32 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractProjectParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.FlowParser; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -/** - * Created by allenlliu on 2019/9/16. - */ - -public class AzkabanProjectParser extends AbstractProjectParser { - - public AzkabanProjectParser(){ - FlowParser[] flowParsers = {new AzkabanFlowParser()}; - setFlowParsers(flowParsers); - } - - private static final Logger logger = LoggerFactory.getLogger(AzkabanProjectParser.class); - - @Override - protected AbstractSchedulerProject createSchedulerProject() { - return new AzkabanSchedulerProject(); - } - - @Override - public void setFlowParsers(FlowParser[] flowParsers) { - super.setFlowParsers(flowParsers); - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/LinkisAzkabanNodeParser.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/LinkisAzkabanNodeParser.java deleted file mode 100644 index e9c8b86a9..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/LinkisAzkabanNodeParser.java +++ /dev/null @@ -1,31 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanSchedulerNode; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -public class LinkisAzkabanNodeParser extends AzkabanNodeParser { - - private static final Logger LOGGER = LoggerFactory.getLogger(LinkisAzkabanNodeParser.class); - - - @Override - public SchedulerNode parseNode(DSSNode dssNode) { - LinkisAzkabanSchedulerNode schedulerNode = new LinkisAzkabanSchedulerNode(); - schedulerNode.setDssNode(dssNode); - return schedulerNode; - } - - @Override - public Boolean ifNodeCanParse(DSSNode dssNode) { - //预留 - return true; - } - - @Override - public int getOrder() { - return 1; - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/LinkisAzkabanSendEmailNodeParser.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/LinkisAzkabanSendEmailNodeParser.java deleted file mode 100644 index 723b35f7e..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/parser/LinkisAzkabanSendEmailNodeParser.java +++ /dev/null @@ -1,27 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanReadNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanSchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.SendEmailNodeParser; - - - -public class LinkisAzkabanSendEmailNodeParser extends SendEmailNodeParser { - - @Override - protected ReadNode createReadNode() { - return new LinkisAzkabanReadNode(); - } - - @Override - protected SchedulerNode createSchedulerNode() { - return new LinkisAzkabanSchedulerNode(); - } - - @Override - public int getOrder() { - return 2; - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanProjectService.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanProjectService.java deleted file mode 100644 index eb536b14b..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanProjectService.java +++ /dev/null @@ -1,223 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.service; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.service.SchedulerProjectService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.project.Project; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.util.AzkabanUtils; -import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; -import com.webank.wedatasphere.dss.common.utils.ZipHelper; -import org.apache.commons.io.IOUtils; -import org.apache.commons.lang.StringUtils; -import org.apache.http.Consts; -import org.apache.http.Header; -import org.apache.http.HttpEntity; -import org.apache.http.NameValuePair; -import org.apache.http.client.CookieStore; -import org.apache.http.client.entity.EntityBuilder; -import org.apache.http.client.entity.UrlEncodedFormEntity; -import org.apache.http.client.methods.CloseableHttpResponse; -import org.apache.http.client.methods.HttpGet; -import org.apache.http.client.methods.HttpPost; -import org.apache.http.client.protocol.HttpClientContext; -import org.apache.http.cookie.Cookie; -import org.apache.http.entity.ContentType; -import org.apache.http.entity.mime.MultipartEntityBuilder; -import org.apache.http.impl.client.BasicCookieStore; -import org.apache.http.impl.client.CloseableHttpClient; -import org.apache.http.impl.client.HttpClients; -import org.apache.http.message.BasicNameValuePair; -import org.apache.http.protocol.HTTP; -import org.apache.http.util.EntityUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.File; -import java.io.InputStream; -import java.io.UnsupportedEncodingException; -import java.net.URLDecoder; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; -import java.util.Optional; - -/** - * Created by cooperyang on 2019/9/16. - */ -public final class AzkabanProjectService extends AppJointUrlImpl implements SchedulerProjectService { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - private String projectUrl; - - @Override - public void setBaseUrl(String baseUrl) { - this.projectUrl = baseUrl + "/manager"; - } - - @Override - public List fetchProjects(Session session) { - return null; - } - - /** - * parameters: - * name = value - * description=value - * - * @param project - * @param session - * @throws AppJointErrorException - */ - @Override - public Project createProject(Project project, Session session) throws AppJointErrorException { - List params = new ArrayList<>(); - params.add(new BasicNameValuePair("action", "create")); - params.add(new BasicNameValuePair("name", project.getName())); - params.add(new BasicNameValuePair("description", project.getDescription())); - HttpPost httpPost = new HttpPost(projectUrl); - httpPost.addHeader(HTTP.CONTENT_ENCODING, HTTP.IDENTITY_CODING); - CookieStore cookieStore = new BasicCookieStore(); - cookieStore.addCookie(session.getCookies()[0]); - HttpEntity entity = EntityBuilder.create() - .setContentType(ContentType.create("application/x-www-form-urlencoded", Consts.UTF_8)) - .setParameters(params).build(); - httpPost.setEntity(entity); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpPost); - HttpEntity ent = response.getEntity(); - String entStr = IOUtils.toString(ent.getContent(), "utf-8"); - logger.error("新建工程 {}, azkaban 返回的信息是 {}", project.getName(), entStr); - String message = AzkabanUtils.handleAzkabanEntity(entStr); - if (!"success".equals(message)) { - throw new AppJointErrorException(90008, "新建工程失败, 原因:" + message); - } - } catch (Exception e) { - logger.error("创建工程失败:", e); - throw new AppJointErrorException(90009, e.getMessage(), e); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - return null; - } - - /** - * delete=boolean - * project=projectName - * - * @param project - * @param session - */ - @Override - public void deleteProject(Project project, Session session) throws AppJointErrorException { - List params = new ArrayList<>(); - params.add(new BasicNameValuePair("delete", "true")); - params.add(new BasicNameValuePair("project", project.getName())); - CookieStore cookieStore = new BasicCookieStore(); - cookieStore.addCookie(session.getCookies()[0]); - HttpClientContext context = HttpClientContext.create(); - CloseableHttpResponse response = null; - CloseableHttpClient httpClient = null; - try { - String finalUrl = projectUrl + "?" + EntityUtils.toString(new UrlEncodedFormEntity(params)); - HttpGet httpGet = new HttpGet(finalUrl); - httpGet.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpGet, context); - Header[] allHeaders = context.getRequest().getAllHeaders(); - Optional
header = Arrays.stream(allHeaders).filter(f -> "Cookie".equals(f.getName())).findFirst(); - header.ifPresent(DSSExceptionUtils.handling(this::parseCookie)); - } catch (Exception e) { - logger.error("delete scheduler project failed,reason:",e); - throw new AppJointErrorException(90010, e.getMessage(), e); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - } - - private void parseCookie(Header header) { - String[] arr = header.getValue().split(";"); - Arrays.stream(arr).filter(f -> f.toLowerCase().trim().contains("azkaban.failure.message=")) - .forEach(DSSExceptionUtils.handling(this::parseHeader)); - } - - private void parseHeader(String headerValue) throws UnsupportedEncodingException, AppJointErrorException { - String errorMsg = URLDecoder.decode(headerValue.split("=")[1], "UTF-8"); - if (StringUtils.isNotEmpty(errorMsg)) { - throw new AppJointErrorException(90011, errorMsg); - } - } - - @Override - public void publishProject(Project project, Session session) throws AppJointErrorException { - String tmpSavePath; - try { - tmpSavePath = zipProject(project); - //upload zip to Azkaban - uploadProject(tmpSavePath,project,session.getCookies()[0]); - } catch (Exception e) { - logger.error("upload sheduler failed:reason",e); - throw new AppJointErrorException(90012, e.getMessage(), e); - } - } - - private void uploadProject(String tmpSavePath, Project project, Cookie cookie) throws AppJointErrorException { - String projectName = project.getName(); - HttpPost httpPost = new HttpPost(projectUrl + "?project=" + projectName); - httpPost.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - CloseableHttpResponse response = null; - File file = new File(tmpSavePath); - CookieStore cookieStore = new BasicCookieStore(); - cookieStore.addCookie(cookie); - CloseableHttpClient httpClient = null; - InputStream inputStream = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - MultipartEntityBuilder entityBuilder = MultipartEntityBuilder.create(); - entityBuilder.addBinaryBody("file",file); - entityBuilder.addTextBody("ajax", "upload"); - entityBuilder.addTextBody("project", projectName); - httpPost.setEntity(entityBuilder.build()); - response = httpClient.execute(httpPost); - HttpEntity httpEntity = response.getEntity(); - inputStream = httpEntity.getContent(); - String entStr = null; - entStr = IOUtils.toString(inputStream, "utf-8"); - if(response.getStatusLine().getStatusCode() != 200){ - logger.error("调用azkaban上传接口的返回不为200, status code 是 {}", response.getStatusLine().getStatusCode()); - throw new AppJointErrorException(90013, "release project failed, " + entStr); - } - logger.info("upload project:{} success!",projectName); - }catch (Exception e){ - logger.error("upload failed,reason:",e); - throw new AppJointErrorException(90014,e.getMessage(), e); - } - finally { - IOUtils.closeQuietly(inputStream); - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - } - - @Override - public void updateProject(Project project, Session session) throws AppJointErrorException { - - } - - public String zipProject(Project project) throws DSSErrorException { - AzkabanSchedulerProject publishProject = (AzkabanSchedulerProject) project; - String projectPath = publishProject.getStorePath(); - String zipPath = ZipHelper.zip(projectPath); - return zipPath; - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanSecurityService.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanSecurityService.java deleted file mode 100644 index b24fe94e7..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanSecurityService.java +++ /dev/null @@ -1,160 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.service; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.conf.AzkabanConf; -import com.webank.wedatasphere.dss.appjoint.scheduler.service.SchedulerSecurityService; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.constant.AzkabanConstant; -import com.webank.wedatasphere.linkis.common.utils.Utils; -import org.apache.commons.io.IOUtils; -import org.apache.http.HttpEntity; -import org.apache.http.NameValuePair; -import org.apache.http.client.CookieStore; -import org.apache.http.client.entity.UrlEncodedFormEntity; -import org.apache.http.client.methods.CloseableHttpResponse; -import org.apache.http.client.methods.HttpPost; -import org.apache.http.client.protocol.HttpClientContext; -import org.apache.http.cookie.Cookie; -import org.apache.http.impl.client.BasicCookieStore; -import org.apache.http.impl.client.CloseableHttpClient; -import org.apache.http.impl.client.HttpClients; -import org.apache.http.message.BasicNameValuePair; -import org.apache.http.util.EntityUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.IOException; -import java.util.ArrayList; -import java.util.List; -import java.util.Optional; -import java.util.Properties; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.TimeUnit; - -/** - * Created by cooperyang on 2019/9/16. - */ -public final class AzkabanSecurityService extends AppJointUrlImpl implements SchedulerSecurityService { - - private static Logger LOGGER = LoggerFactory.getLogger(AzkabanSecurityService.class); - - private static final long EXPIRE_TIME = 1000 * 60 * 10L; - private ConcurrentHashMap sessionCache = new ConcurrentHashMap<>(); - private String securityUrl; - private static final String USER_NAME_KEY = "username"; - private static final String USER_TOKEN_KEY = AzkabanConf.AZKABAN_LOGIN_PWD.getValue(); - private static final String SESSION_ID_KEY = "azkaban.browser.session.id"; - private static Properties userToken ; - - static { - Utils.defaultScheduler().scheduleAtFixedRate(()->{ - LOGGER.info("load azkaban-user.xml"); - Properties properties = new Properties(); - try { - properties.load(AzkabanSecurityService.class.getClassLoader().getResourceAsStream(AzkabanConstant.TOKEN_FILE_NAME)); - userToken = properties; - } catch (IOException e) { - LOGGER.error("load error:",e); - } - },0,10, TimeUnit.MINUTES); - } - - public void reloadToken(){ - LOGGER.info("reload azkaban-user.xml"); - Properties properties = new Properties(); - try { - properties.load(AzkabanSecurityService.class.getClassLoader().getResourceAsStream(AzkabanConstant.TOKEN_FILE_NAME)); - userToken = properties; - } catch (IOException e) { - LOGGER.error("reload error:",e); - } - } - - @Override - public void setBaseUrl(String baseUrl) { - this.securityUrl = baseUrl + "/checkin"; - } - - @Override - public Session login(String user) throws AppJointErrorException { - synchronized (user.intern()) { - Session session = sessionCache.get(user); - if (session != null && !isExpire(session)) { - session.updateLastAccessTime(); - return session; - } - Session newSession; - try { - newSession = getSession(user, getUserToken(user)); - } catch (Exception e) { - LOGGER.error("获取session失败:", e); - throw new AppJointErrorException(90019, e.getMessage(), e); - } - ((AzkabanSession) newSession).setUser(user); - sessionCache.put(user, newSession); - return newSession; - } - } - - private Session getSession(String user, String token) throws IOException, AppJointErrorException { - HttpPost httpPost = new HttpPost(securityUrl); - List params = new ArrayList<>(); - params.add(new BasicNameValuePair(USER_NAME_KEY, user)); - params.add(new BasicNameValuePair(USER_TOKEN_KEY, token)); - params.add(new BasicNameValuePair("action", "login")); - httpPost.setEntity(new UrlEncodedFormEntity(params)); - CookieStore cookieStore = new BasicCookieStore(); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - HttpClientContext context; - String responseContent; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - context = HttpClientContext.create(); - response = httpClient.execute(httpPost, context); - HttpEntity entity = response.getEntity(); - responseContent = EntityUtils.toString(entity,"utf-8"); - LOGGER.info("Get azkaban response code is "+ response.getStatusLine().getStatusCode()+",response: "+responseContent); - if(response.getStatusLine().getStatusCode() != 200){ - throw new AppJointErrorException(90041, responseContent); - } - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - List cookies = context.getCookieStore().getCookies(); - Optional session = cookies.stream().filter(this::findSessionId).map(this::cookieToSession).findFirst(); - return session.orElseThrow(() -> new AppJointErrorException(90041,"Get azkaban session is null : "+ responseContent)); - } - - private boolean findSessionId(Cookie cookie) { - return SESSION_ID_KEY.equals(cookie.getName()); - } - - private Session cookieToSession(Cookie cookie) { - AzkabanSession session = new AzkabanSession(); - session.setCookies(new Cookie[]{cookie}); - return session; - } - - private String getUserToken(String user) throws AppJointErrorException { - //直接从配置文件中读取，有需求可以自己实现 - Object token = userToken.get(user); - if (token == null) throw new AppJointErrorException(90020,"用户token为空"); - return token.toString(); - } - - @Override - public void logout(String user) { - synchronized (user.intern()) { - sessionCache.remove(user); - } - } - - private boolean isExpire(Session session) { - return System.currentTimeMillis() - session.getLastAccessTime() > EXPIRE_TIME; - } - - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanSession.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanSession.java deleted file mode 100644 index 77219cecf..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/service/AzkabanSession.java +++ /dev/null @@ -1,59 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.service; - -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import org.apache.http.Header; -import org.apache.http.cookie.Cookie; - -import java.util.HashMap; -import java.util.Map; - -/** - * Created by cooperyang on 2019/9/16. - */ -public final class AzkabanSession implements Session { - - private long lastAccessTime; - private Cookie[] cookies; - private Header[] headers; - private String user; - private Map parameters = new HashMap<>(); - - @Override - public String getUser() { - return user; - } - - public void setUser(String user) { - this.user = user; - } - - @Override - public Cookie[] getCookies() { - return this.cookies; - } - - @Override - public Header[] getHeaders() { - return this.headers; - } - - @Override - public Map getParameters() { - return this.parameters; - } - - @Override - public long getLastAccessTime() { - return this.lastAccessTime; - } - - public void setCookies(Cookie[] cookies){ - this.cookies = cookies; - } - - @Override - public void updateLastAccessTime() { - lastAccessTime = System.currentTimeMillis(); - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/AzkabanProjectTuning.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/AzkabanProjectTuning.java deleted file mode 100644 index b4cec7a03..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/AzkabanProjectTuning.java +++ /dev/null @@ -1,69 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.conf.AzkabanConf; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.AbstractProjectTuning; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.FlowTuning; -import java.io.File; -import java.text.SimpleDateFormat; -import java.util.ArrayList; -import java.util.Date; -import java.util.List; - -/** - * Created by cooperyang on 2019/9/26. - */ - -public class AzkabanProjectTuning extends AbstractProjectTuning { - - public AzkabanProjectTuning(){ - ArrayList list = new ArrayList<>(); - list.add(new LinkisAzkabanFlowTuning()); - list.add(new LinkisShareNodeFlowTuning()); - FlowTuning[] flowTunings =new FlowTuning[list.size()]; - setFlowTunings(list.toArray(flowTunings)); - } - - @Override - public SchedulerProject tuningSchedulerProject(SchedulerProject schedulerProject) { - if(ifProjectCanTuning(schedulerProject)){ - AzkabanSchedulerProject project = (AzkabanSchedulerProject) schedulerProject; - //1.给AzkabanSchedulerProject的storepath赋值 - assignStorePath(project); - //2.给他的rootflow的storepath也赋值 - List schedulerFlows = project.getSchedulerFlows(); - schedulerFlows.forEach(flow ->setRootFlowStorePath(project.getStorePath(),flow)); - } - return super.tuningSchedulerProject(schedulerProject); - } - - public void setRootFlowStorePath(String projectStorePath,SchedulerFlow schedulerFlow){ - AzkabanSchedulerFlow azkabanSchedulerFlow = (AzkabanSchedulerFlow) schedulerFlow; - azkabanSchedulerFlow.setStorePath(projectStorePath + File.separator + azkabanSchedulerFlow.getName()); - } - - - @Override - public void setFlowTunings(FlowTuning[] flowTunings) { - super.setFlowTunings(flowTunings); - } - - - public boolean ifProjectCanTuning(SchedulerProject schedulerProject){ - return schedulerProject instanceof AzkabanSchedulerProject; - } - - private void assignStorePath(AzkabanSchedulerProject azkabanSchedulerProject) { - SimpleDateFormat dateFormat = new SimpleDateFormat(AzkabanSchedulerProject.DATE_FORMAT); - Date date = new Date(); - String dataStr = dateFormat.format(date); - String userName = azkabanSchedulerProject.getDssProject().getUserName(); - String name = azkabanSchedulerProject.getName(); - String storePath = AzkabanConf.DEFAULT_STORE_PATH.getValue() + File.separator + userName - + File.separator + dataStr + File.separator +name; - azkabanSchedulerProject.setStorePath(storePath); - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/LinkisAzkabanFlowTuning.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/LinkisAzkabanFlowTuning.java deleted file mode 100644 index 42c9962f1..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/LinkisAzkabanFlowTuning.java +++ /dev/null @@ -1,98 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.tuning; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanSchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerEdge; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.AbstractFlowTuning; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.NodeTuning; -import com.webank.wedatasphere.dss.common.entity.node.DSSNodeDefault; -import java.io.File; -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - -//DefaultFlowTuning修改为AzkabanFlowTuning -public class LinkisAzkabanFlowTuning extends AbstractFlowTuning { - - - - @Override - public SchedulerFlow tuningSchedulerFlow(SchedulerFlow schedulerFlow) { - //1.尾部添加依赖 - //2.设置子flow和node存储路径 - SchedulerFlow addEndNodeSchedulerFlow = addEndNodeForFlowName(schedulerFlow); - String flowStorePath = ((AzkabanSchedulerFlow) addEndNodeSchedulerFlow).getStorePath(); - if (addEndNodeSchedulerFlow.getChildren() != null) { - addEndNodeSchedulerFlow.getChildren().forEach(flow -> setFlowStorePath(flowStorePath, flow)); - } - - if (addEndNodeSchedulerFlow.getSchedulerNodes() != null) { - addEndNodeSchedulerFlow.getSchedulerNodes().forEach(node -> setNodeStorePath(flowStorePath, node)); - } - return super.tuningSchedulerFlow(schedulerFlow); - } - - private void setFlowStorePath(String flowStorePath, SchedulerFlow schedulerFlow) { - AzkabanSchedulerFlow azkabanSchedulerFlow = (AzkabanSchedulerFlow) schedulerFlow; - azkabanSchedulerFlow.setStorePath(flowStorePath + File.separator + "subFlows" + File.separator + azkabanSchedulerFlow.getName()); - } - - private void setNodeStorePath(String flowStorePath, SchedulerNode schedulerNode) { - AzkabanSchedulerNode azkabanSchedulerNode = (AzkabanSchedulerNode) schedulerNode; - azkabanSchedulerNode.setStorePath(flowStorePath + File.separator + "jobs" + File.separator + schedulerNode.getName()); - } - - @Override - public NodeTuning[] getNodeTunings() { - return new NodeTuning[0]; - } - - @Override - public Boolean ifFlowCanTuning(SchedulerFlow schedulerFlow) { - // TODO: 2019/9/30 应该加上node 是linkisazkabanNode的 instance判断，因为加到尾部节点的就是一个linkisazkabanNode - return schedulerFlow instanceof AzkabanSchedulerFlow; - } - - private SchedulerFlow addEndNodeForFlowName(SchedulerFlow flow) { - DSSNodeDefault endNode = new DSSNodeDefault(); - List endNodeList = getFlowEndJobList(flow); - endNode.setId(flow.getName() + "_"); - endNode.setName(flow.getName() + "_"); - endNode.setNodeType("linkis.control.empty"); - Map jobContentMap = new java.util.HashMap(); - /*jobContentMap.put("script", "");*/ - endNode.setJobContent(jobContentMap); - if (endNodeList.size() > 0) { - endNodeList.forEach(tmpNode -> endNode.addDependency(tmpNode.getName())); - } - LinkisAzkabanSchedulerNode azkabanSchedulerNode = new LinkisAzkabanSchedulerNode(); - azkabanSchedulerNode.setDssNode(endNode); - flow.getSchedulerNodes().add((azkabanSchedulerNode)); - return flow; - } - - private List getFlowEndJobList(SchedulerFlow flow) { - List res = new ArrayList<>(); - for (SchedulerNode job : flow.getSchedulerNodes()) { - int flag = 0; - for (SchedulerEdge link : flow.getSchedulerEdges()) { - if (job.getId().equals(link.getDssEdge().getSource())) { - flag = 1; - } - } - if (flag == 0) { - res.add(job); - } - } - return res; - } - - @Override - public int getOrder() { - return 1; - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/LinkisShareNodeFlowTuning.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/LinkisShareNodeFlowTuning.java deleted file mode 100644 index 2e01c2582..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/tuning/LinkisShareNodeFlowTuning.java +++ /dev/null @@ -1,34 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.tuning; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.AzkabanSchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.entity.LinkisAzkabanShareNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.AbstractShareNodeFlowTuning; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.NodeTuning; - - - -public class LinkisShareNodeFlowTuning extends AbstractShareNodeFlowTuning { - - @Override - public Boolean ifFlowCanTuning(SchedulerFlow schedulerFlow) { - return schedulerFlow instanceof AzkabanSchedulerFlow; - } - - @Override - public int getOrder() { - return 2; - } - - @Override - protected ShareNode createShareNode() { - return new LinkisAzkabanShareNode(); - } - - @Override - public NodeTuning[] getNodeTunings() { - return new NodeTuning[0]; - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/util/AzkabanUtils.java b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/util/AzkabanUtils.java deleted file mode 100644 index 3fb7801b3..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/util/AzkabanUtils.java +++ /dev/null @@ -1,34 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.util; - -import com.google.gson.Gson; -import org.apache.commons.lang.StringUtils; - -import java.io.IOException; -import java.util.Map; - -/** - * Created by cooperyang on 2019/9/18. - */ -public class AzkabanUtils { - public static String handleAzkabanEntity(String entityString) throws IOException { - Gson gson = new Gson(); - Object object = gson.fromJson(entityString, Object.class); - String status = null; - String message = null; - if (object instanceof Map) { - Map map = (Map) object; - if (map.get("status") != null) { - status = map.get("status").toString(); - } - if (StringUtils.isNotEmpty(status)) { - if (null != map.get("message")) { - message = map.get("message").toString(); - } - } - if ("error".equalsIgnoreCase(status)) { - return message; - } - } - return "success"; - } -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/util/AzkabanUtilsScala.scala b/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/util/AzkabanUtilsScala.scala deleted file mode 100644 index 579f133b4..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/azkaban/util/AzkabanUtilsScala.scala +++ /dev/null @@ -1,17 +0,0 @@ -package com.webank.wedatasphere.dss.appjoint.scheduler.azkaban.util - -import java.util -import scala.collection.JavaConversions._ -import scala.collection.JavaConverters._ - -/** - * Created by cooperyang on 2019/9/27. - */ -object AzkabanUtilsScala { - - def getRepeatNodeName(nodeList:java.util.List[String]):util.List[String]={ - val res = nodeList.map(x=>(x,1)).groupBy(x => x._1).map(x =>(x._1,x._2.size)).filter(x=>x._2 >1).map(x=>x._1).toList - res.asJava - } - -} diff --git a/dss-azkaban-scheduler-appjoint/src/main/resources/appjoint.properties b/dss-azkaban-scheduler-appjoint/src/main/resources/appjoint.properties deleted file mode 100644 index e5983a459..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/resources/appjoint.properties +++ /dev/null @@ -1,21 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - - - - - diff --git a/dss-azkaban-scheduler-appjoint/src/main/resources/log4j.properties b/dss-azkaban-scheduler-appjoint/src/main/resources/log4j.properties deleted file mode 100644 index 0807e6087..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/resources/log4j.properties +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO diff --git a/dss-azkaban-scheduler-appjoint/src/main/resources/log4j2.xml b/dss-azkaban-scheduler-appjoint/src/main/resources/log4j2.xml deleted file mode 100644 index 3923cd9f3..000000000 --- a/dss-azkaban-scheduler-appjoint/src/main/resources/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dss-common/pom.xml b/dss-common/pom.xml deleted file mode 100644 index 4fe637f11..000000000 --- a/dss-common/pom.xml +++ /dev/null @@ -1,67 +0,0 @@ - - - - - 4.0.0 - - dss - com.webank.wedatasphere.dss - 0.9.1 - - dss-common - - - - com.webank.wedatasphere.linkis - linkis-common - ${linkis.version} - - - - - - - - org.apache.maven.plugins - maven-compiler-plugin - 3.3 - - 1.8 - 1.8 - UTF-8 - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - - - \ No newline at end of file diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlow.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlow.java deleted file mode 100644 index ff768f791..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlow.java +++ /dev/null @@ -1,194 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.flow; - -import java.util.Date; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/5/14. - */ -public class DSSFlow implements Flow { - private Long id; - private String name; - private Boolean state; //0,1代表发布过和未发布过 - private String source; - private String description; - private Date createTime; - private Long creatorID; - private Boolean isRootFlow; - private Integer rank; - private Long projectID; - private Boolean hasSaved;//0disable 1 enable 0表示工作流从来没存过，发布的时候忽略 - private String uses; - - private List versions; //为了前台不做修改，还是使用versions 而不使用flowVersions的变量名 - private List children; - private String flowType; - - private DSSFlowVersion latestVersion; - - - public Integer getRank() { - return rank; - } - - public void setRank(Integer rank) { - this.rank = rank; - } - - @Override - public Long getId() { - return id; - } - - @Override - public void setId(Long id) { - this.id = id; - } - - @Override - public String getName() { - return name; - } - - @Override - public void setName(String name) { - this.name = name; - } - - - @Override - public String getDescription() { - return description; - } - - @Override - public void setDescription(String description) { - this.description = description; - } - - @Override - public void addFlowVersion(FlowVersion flowVersion) { - this.versions.add((DSSFlowVersion) flowVersion); - } - - @Override - public List getChildren() { - return children; - } - - @Override - public void setChildren(List children) { - this.children = children.stream().map(f ->(DSSFlow)f).collect(Collectors.toList()); - } - - @Override - public List getFlowVersions() { - return this.versions; - } - - @Override - public void setFlowVersions(List flowVersions) { - this.versions = flowVersions.stream().map(f ->(DSSFlowVersion)f).collect(Collectors.toList()); - } - - @Override - public String getFlowType() { - return flowType; - } - - @Override - public void setFlowType(String flowType) { - this.flowType = flowType; - } - - @Override - public Boolean getRootFlow() { - return isRootFlow; - } - - public void setRootFlow(Boolean rootFlow) { - isRootFlow = rootFlow; - } - - public Boolean getState() { - return state; - } - - public void setState(Boolean state) { - this.state = state; - } - - public String getSource() { - return source; - } - - public void setSource(String source) { - this.source = source; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Long getCreatorID() { - return creatorID; - } - - public void setCreatorID(Long creatorID) { - this.creatorID = creatorID; - } - - public Long getProjectID() { - return projectID; - } - - public void setProjectID(Long projectID) { - this.projectID = projectID; - } - - public Boolean getHasSaved() { - return hasSaved; - } - - public void setHasSaved(Boolean hasSaved) { - this.hasSaved = hasSaved; - } - - public DSSFlowVersion getLatestVersion() { - return latestVersion; - } - - public void setLatestVersion(DSSFlowVersion latestVersion) { - this.latestVersion = latestVersion; - } - - public String getUses() { - return uses; - } - - public void setUses(String uses) { - this.uses = uses; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlowPublishHistory.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlowPublishHistory.java deleted file mode 100644 index 7fe3e9ad4..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlowPublishHistory.java +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.flow; - -/** - * Created by enjoyyin on 2019/9/19. - */ -public class DSSFlowPublishHistory { -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlowVersion.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlowVersion.java deleted file mode 100644 index ce8559097..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSFlowVersion.java +++ /dev/null @@ -1,163 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.flow; - -import java.util.Date; - -/** - * Created by enjoyyin on 2019/9/19. - */ -public class DSSFlowVersion implements FlowVersion, Comparable { - private Long id; - private Long flowID; - private String source; - private String jsonPath; - private String comment; - private Date updateTime; - private Long updatorID; - private String version; - private DSSFlowPublishHistory publishHistory; - private String json; - private String updator; - private Boolean isNotPublished; //true 未发过版，false已经过版 - private Long projectVersionID; - private Long oldFlowID; - - public Long getOldFlowID() { - return oldFlowID; - } - - public void setOldFlowID(Long oldFlowID) { - this.oldFlowID = oldFlowID; - } - - public Long getProjectVersionID() { - return projectVersionID; - } - - public void setProjectVersionID(Long projectVersionID) { - this.projectVersionID = projectVersionID; - } - - public Boolean getNotPublished() { - return isNotPublished; - } - - public void setNotPublished(Boolean notPublished) { - isNotPublished = notPublished; - } - - public String getUpdator() { - return updator; - } - - public void setUpdator(String updator) { - this.updator = updator; - } - - public String getJson() { - return json; - } - - public void setJson(String json) { - this.json = json; - } - - @Override - public String getVersion() { - return version; - } - - @Override - public void setVersion(String version) { - this.version = version; - } - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public Long getFlowID() { - return flowID; - } - - public void setFlowID(Long flowID) { - this.flowID = flowID; - } - - public String getSource() { - return source; - } - - public void setSource(String source) { - this.source = source; - } - - public String getJsonPath() { - return jsonPath; - } - - public void setJsonPath(String jsonPath) { - this.jsonPath = jsonPath; - } - - @Override - public String getComment() { - return comment; - } - - @Override - public void setComment(String comment) { - this.comment = comment; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Long getUpdatorID() { - return updatorID; - } - - public void setUpdatorID(Long updatorID) { - this.updatorID = updatorID; - } - - public DSSFlowPublishHistory getPublishHistory() { - return publishHistory; - } - - public void setPublishHistory(DSSFlowPublishHistory publishHistory) { - this.publishHistory = publishHistory; - } - - @Override - public int compareTo(DSSFlowVersion o) { - Integer v1 = Integer.valueOf(this.version.substring(1, version.length())); - Integer v2 = Integer.valueOf(o.version.substring(1,o.version.length())); - return v2 - v1; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSJSONFlow.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSJSONFlow.java deleted file mode 100644 index ae9b98631..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/DSSJSONFlow.java +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.flow; - -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/5/14. - */ -public class DSSJSONFlow extends DSSFlow { - private String json; - - private List children; - - public String getJson() { - return json; - } - - public void setJson(String json) { - this.json = json; - } - - @Override - public void setChildren(List children) { - this.children = children.stream().map(f ->(DSSJSONFlow)f).collect(Collectors.toList()); - } - - @Override - public List getChildren() { - return children; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/Flow.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/Flow.java deleted file mode 100644 index d986c025d..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/Flow.java +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.flow; - -import java.util.List; - -/** - * Created by enjoyyin on 2019/5/14. - */ -public interface Flow { - Long getId(); - - void setId(Long id); - - String getName(); - - void setName(String name); - - String getFlowType(); - - void setFlowType(String flowType); - - String getDescription(); - - void setDescription(String description); - - Boolean getRootFlow(); - - List getChildren(); - - void setChildren(List children); - - List getFlowVersions(); - - void setFlowVersions(List flowVersions); - - void addFlowVersion(FlowVersion flowVersion); -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSJSONProject.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSJSONProject.java deleted file mode 100644 index ffd9026fc..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSJSONProject.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSJSONFlow; - -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by allenlliu on 2019/9/16. - */ -public class DSSJSONProject extends DSSProject { - private List flows; - - @Override - public List getFlows() { - return this.flows; - } - - @Override - public void setFlows(List flows) { - this.flows = flows.stream().map(f ->(DSSJSONFlow)f).collect(Collectors.toList()); - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProject.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProject.java deleted file mode 100644 index 4454362d8..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProject.java +++ /dev/null @@ -1,294 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -import com.webank.wedatasphere.dss.common.entity.Resource; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; - -import java.util.Date; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public class DSSProject implements Project { - - private Long id; - private String name; - private String description; - private String source; - private Long userID; - private Date createTime; - private Long createBy; - private Date updateTime; - private Long updateBy; - private Long orgID; - private Boolean visibility; - private Boolean isTransfer; - private Boolean isArchive; - private Long initialOrgID; - private String pic; - private Long starNum; - private String product; - private Integer applicationArea; - private String business; - private Long workspaceId; - - private DSSProjectVersion latestVersion; - private Boolean isNotPublish; - private String userName; - - private String projectGroup; - private List projectVersions; - private List flows; - private List projectResources; - - public List getProjectResources() { - return projectResources; - } - - public void setProjectResources(List projectResources) { - this.projectResources = projectResources; - } - - public List getFlows() { - return flows; - } - - public void setFlows(List flows) { - this.flows = flows.stream().map(f -> (DSSFlow) f).collect(Collectors.toList()); - } - - public String getUserName() { - return userName; - } - - public void setUserName(String userName) { - this.userName = userName; - } - - public Boolean getNotPublish() { - return isNotPublish; - } - - public void setNotPublish(Boolean notPublish) { - isNotPublish = notPublish; - } - - public Long getOrgID() { - return orgID; - } - - public void setOrgID(Long orgID) { - this.orgID = orgID; - } - - public Boolean getVisibility() { - return visibility; - } - - public void setVisibility(Boolean visibility) { - this.visibility = visibility; - } - - public Boolean getTransfer() { - return isTransfer; - } - - public void setTransfer(Boolean transfer) { - isTransfer = transfer; - } - - public Long getInitialOrgID() { - return initialOrgID; - } - - public void setInitialOrgID(Long initialOrgID) { - this.initialOrgID = initialOrgID; - } - - public DSSProjectVersion getLatestVersion() { - return latestVersion; - } - - public void setLatestVersion(DSSProjectVersion latestVersion) { - this.latestVersion = latestVersion; - } - - public String getSource() { - return source; - } - - public void setSource(String source) { - this.source = source; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - - @Override - public String getDescription() { - return description; - } - - @Override - public void setDescription(String description) { - this.description = description; - } - - @Override - public String getPorjectGroup() { - return projectGroup; - } - - @Override - public void setProjectGroup(String projectGroup) { - this.projectGroup = projectGroup; - } - - @Override - public List getProjectVersions() { - return projectVersions; - } - - @Override - public void setProjectVersions(List projectVersions) { - this.projectVersions = projectVersions.stream().map(f -> (DSSProjectVersion) f).collect(Collectors.toList()); - } - - - @Override - public void addProjectVersion(ProjectVersion projectVersion) { - this.projectVersions.add((DSSProjectVersion) projectVersion); - } - - @Override - public Long getId() { - return id; - } - - @Override - public void setId(Long id) { - this.id = id; - } - - @Override - public String getName() { - return name; - } - - @Override - public void setName(String name) { - this.name = name; - } - - public Long getUserID() { - return userID; - } - - public void setUserID(Long userID) { - this.userID = userID; - } - - public Long getCreateBy() { - return createBy; - } - - public void setCreateBy(Long createBy) { - this.createBy = createBy; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Long getUpdateBy() { - return updateBy; - } - - public void setUpdateBy(Long updateBy) { - this.updateBy = updateBy; - } - - public Boolean getArchive() { - return isArchive; - } - - public void setArchive(Boolean archive) { - isArchive = archive; - } - - public String getPic() { - return pic; - } - - public void setPic(String pic) { - this.pic = pic; - } - - public Long getStarNum() { - return starNum; - } - - public void setStarNum(Long starNum) { - this.starNum = starNum; - } - - public String getProduct() { - return product; - } - - public void setProduct(String product) { - this.product = product; - } - - public Integer getApplicationArea() { - return applicationArea; - } - - public void setApplicationArea(Integer applicationArea) { - this.applicationArea = applicationArea; - } - - public String getBusiness() { - return business; - } - - public void setBusiness(String business) { - this.business = business; - } - - public Long getWorkspaceId() { - return workspaceId; - } - - public void setWorkspaceId(Long workspaceId) { - this.workspaceId = workspaceId; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProjectPublishHistory.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProjectPublishHistory.java deleted file mode 100644 index dca93dd48..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProjectPublishHistory.java +++ /dev/null @@ -1,116 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -import java.util.Date; - -/** - * Created by enjoyyin on 2019/5/14. - */ -public class DSSProjectPublishHistory { - private Long id; - private Long projectVersionID; - private Date createTime; - private Long createID; - private String creator; - private Date updateTime; - private String comment; - private Integer state; - private String versionPath; - private Date expireTime; - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public Long getProjectVersionID() { - return projectVersionID; - } - - public void setProjectVersionID(Long projectVersionID) { - this.projectVersionID = projectVersionID; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Long getCreateID() { - return createID; - } - - public void setCreateID(Long createID) { - this.createID = createID; - } - - public String getCreator() { - return creator; - } - - public void setCreator(String creator) { - this.creator = creator; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public String getComment() { - return comment; - } - - public void setComment(String comment) { - this.comment = comment; - } - - public Integer getState() { - return state; - } - - public void setState(Integer state) { - this.state = state; - } - - public String getVersionPath() { - return versionPath; - } - - public void setVersionPath(String versionPath) { - this.versionPath = versionPath; - } - - public Date getExpireTime() { - return expireTime; - } - - public void setExpireTime(Date expireTime) { - this.expireTime = expireTime; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProjectVersion.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProjectVersion.java deleted file mode 100644 index 5731ecf8e..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProjectVersion.java +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -import java.util.Date; - -/** - * Created by enjoyyin on 2019/9/18. - */ -public class DSSProjectVersion implements ProjectVersion { - - private Long id; - private Long projectID; - private String version; - private String comment; - private Date updateTime; - private Long updatorID; - private Integer lock; - private String updator; - private Boolean isNotPublish; - private DSSProjectPublishHistory publishHistory; - - @Override - public String getVersion() { - return this.version; - } - - @Override - public void setVersion() { - this.version = version; - } - - @Override - public String getComment() { - return this.comment; - } - - @Override - public void setComment(String comment) { - this.comment = comment; - } - - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public Long getProjectID() { - return projectID; - } - - public void setProjectID(Long projectID) { - this.projectID = projectID; - } - - public void setVersion(String version) { - this.version = version; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Long getUpdatorID() { - return updatorID; - } - - public void setUpdatorID(Long updatorID) { - this.updatorID = updatorID; - } - - public Integer getLock() { - return lock; - } - - public void setLock(Integer lock) { - this.lock = lock; - } - - public String getUpdator() { - return updator; - } - - public void setUpdator(String updator) { - this.updator = updator; - } - - public Boolean getNotPublish() { - return isNotPublish; - } - - public void setNotPublish(Boolean notPublish) { - isNotPublish = notPublish; - } - - public DSSProjectPublishHistory getPublishHistory() { - return publishHistory; - } - - public void setPublishHistory(DSSProjectPublishHistory publishHistory) { - this.publishHistory = publishHistory; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/Project.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/Project.java deleted file mode 100644 index 492d5d601..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/Project.java +++ /dev/null @@ -1,47 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface Project { - Long getId(); - - void setId(Long id); - - String getName(); - - void setName(String name); - - String getDescription(); - - void setDescription(String description); - - String getPorjectGroup(); - - void setProjectGroup(String projectGroup); - - List getProjectVersions(); - - void setProjectVersions(List projectVersions); - - void addProjectVersion(ProjectVersion projectVersion); -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/ProjectVersion.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/ProjectVersion.java deleted file mode 100644 index b9f3c2634..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/ProjectVersion.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface ProjectVersion { - String getVersion(); - - void setVersion(); - - String getComment(); - - void setComment(String comment); -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/ProjectVersionForFlows.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/ProjectVersionForFlows.java deleted file mode 100644 index 773a22cc6..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/ProjectVersionForFlows.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.entity.project; - -import com.webank.wedatasphere.dss.common.entity.flow.Flow; - -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/26. - */ -public interface ProjectVersionForFlows extends ProjectVersion { - - List getFlows(); - - void setFlows(List flows); - - void addFlow(Flow flow); - - void removeFlow(Flow flow); -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSRuntimeException.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSRuntimeException.java deleted file mode 100644 index 057a2e373..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSRuntimeException.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.exception; - -import com.webank.wedatasphere.linkis.common.exception.WarnException; - -/** - * Created by enjoyyin on 2019/9/24. - */ -public class DSSRuntimeException extends WarnException { - - public DSSRuntimeException(String msg){ - super(100000, msg); - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingConsumer.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingConsumer.java deleted file mode 100644 index 54848a970..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingConsumer.java +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.exception; - -/** - * Created by enjoyyin on 2019/9/20. - */ -@FunctionalInterface -public interface ThrowingConsumer { - void accept(T t) throws E; -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingFunction.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingFunction.java deleted file mode 100644 index f261c11ee..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingFunction.java +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.exception; - -/** - * Created by enjoyyin on 2019/11/13. - */ -@FunctionalInterface -public interface ThrowingFunction { - R accept(T t) throws E; -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSExceptionUtils.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSExceptionUtils.java deleted file mode 100644 index 500910933..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSExceptionUtils.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.utils; - -import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; -import com.webank.wedatasphere.dss.common.exception.ThrowingConsumer; -import com.webank.wedatasphere.dss.common.exception.ThrowingFunction; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.function.Consumer; -import java.util.function.Function; - -/** - * Created by enjoyyin on 2019/9/24. - */ -public class DSSExceptionUtils { - - private static final Logger LOGGER = LoggerFactory.getLogger(DSSExceptionUtils.class); - - public static Consumer handling( - ThrowingConsumer throwingConsumer) { - return i -> { - try { - throwingConsumer.accept(i); - } catch (Exception e) { - LOGGER.error("execute failed,reason:",e); - throw new DSSRuntimeException(e.getMessage()); - } - }; - } - - public static Function map( - ThrowingFunction throwingFunction) { - return i -> { - try { - return throwingFunction.accept(i); - } catch (Exception e) { - LOGGER.error("execute failed,reason:",e); - throw new DSSRuntimeException(e.getMessage()); - } - }; - } -} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ZipHelper.java b/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ZipHelper.java deleted file mode 100644 index 594d5c922..000000000 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ZipHelper.java +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.utils; - - - -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.apache.commons.io.IOUtils; -import org.apache.commons.lang.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.BufferedReader; -import java.io.File; -import java.io.InputStreamReader; -import java.util.ArrayList; -import java.util.List; - -/** - * created by enjoyyin on 2019/6/13 - * Description: - */ -public class ZipHelper { - - private static final Logger logger = LoggerFactory.getLogger(ZipHelper.class); - - private static final String ZIP_CMD = "zip"; - private static final String RECURSIVE = "-r"; - private static final String ZIP_TYPE = ".zip"; - /** - * ZipHelper可以将传入的path进行打包 - * @param dirPath 需要打包的project路径,绝对路径 - * @return 打包之后的zip包全路径 - */ - public static String zip(String dirPath)throws DSSErrorException { - if(!FileHelper.checkDirExists(dirPath)){ - logger.error("{} 不存在, 不能创建zip文件", dirPath); - throw new DSSErrorException(90001,dirPath + " does not exist, can not zip"); - } - //先用简单的方法，调用新进程进行压缩 - String[] strArr = dirPath.split(File.separator); - String shortPath = strArr[strArr.length - 1]; - String workPath = dirPath.substring(0, dirPath.length() - shortPath.length() - 1); - List list = new ArrayList<>(); - list.add(ZIP_CMD); - list.add(RECURSIVE); - String zipFilePath = shortPath + ZIP_TYPE; - String longZipFilePath = dirPath + ZIP_TYPE; - list.add(zipFilePath); - list.add(shortPath); - ProcessBuilder processBuilder = new ProcessBuilder(list); - processBuilder.redirectErrorStream(true); - processBuilder.directory(new File(workPath)); - BufferedReader infoReader = null; - BufferedReader errorReader = null; - try{ - Process process = processBuilder.start(); - infoReader = new BufferedReader(new InputStreamReader(process.getInputStream())); - errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream())); - String infoLine = null; - while((infoLine = infoReader.readLine()) != null){ - logger.info("process output: {} ", infoLine); - } - String errorLine = null; - StringBuilder errMsg = new StringBuilder(); - while((errorLine = errorReader.readLine()) != null){ - if (StringUtils.isNotEmpty(errorLine)){ - errMsg.append(errorLine).append("\n"); - } - logger.error("process error: {} ", errorLine); - } - int exitCode = process.waitFor(); - if (exitCode != 0){ - throw new DSSErrorException(90002,errMsg.toString()); - } - }catch(final Exception e){ - logger.error("{} 压缩成 zip 文件失败, reason: ", e); - DSSErrorException exception = new DSSErrorException(90003,dirPath + " to zip file failed"); - exception.initCause(e); - throw exception; - } finally { - //删掉整个目录 - File file = new File(dirPath); - logger.info("开始删除目录 {}", dirPath); - if (deleteDir(file)){ - logger.info("结束删除目录 {} 成功", dirPath); - }else{ - logger.info("删除目录 {} 失败", dirPath); - } - IOUtils.closeQuietly(infoReader); - IOUtils.closeQuietly(errorReader); - } - return longZipFilePath; - } - - private static boolean deleteDir(File dir) { - if (dir.isDirectory()) { - String[] children = dir.list(); - if (children != null && children.length > 0){ - for (String s : children) { - boolean success = deleteDir(new File(dir, s)); - if (!success) { - return false; - } - } - } - } - // 目录此时为空，可以删除 - return dir.delete(); - } -} diff --git a/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/RequestDSSProject.scala b/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/RequestDSSProject.scala deleted file mode 100644 index b9b3da46a..000000000 --- a/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/RequestDSSProject.scala +++ /dev/null @@ -1,25 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.common.protocol - -/** - * Created by enjoyyin on 2019/11/8. - */ -case class RequestDSSProject(flowId:Long, version:String, projectVersionId:Long) - -case class RequestDSSApplication(name:String) \ No newline at end of file diff --git a/dss-commons/dss-common/pom.xml b/dss-commons/dss-common/pom.xml new file mode 100644 index 000000000..1a91fcd17 --- /dev/null +++ b/dss-commons/dss-common/pom.xml @@ -0,0 +1,116 @@ + + + + + 4.0.0 + + dss-commons + com.webank.wedatasphere.dss + 1.0.0 + + dss-common + + + + com.webank.wedatasphere.linkis + linkis-common + + + com.google.code.gson + gson + + + com.fasterxml.jackson.core + jackson-databind + + + org.codehaus.jackson + jackson-mapper-asl + + + org.scala-lang + scala-library + + + org.scala-lang + scala-compiler + + + org.scala-lang + scala-reflect + + + org.scala-lang + scalap + + + commons-lang + commons-lang + + + com.webank.wedatasphere.linkis + linkis-label-common + ${linkis.version} + + + com.webank.wedatasphere.linkis + linkis-protocol + ${linkis.version} + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + + + + product + + 1.0.0-SNAPSHOT + + + + + + + \ No newline at end of file diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/conf/DSSCommonConf.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/conf/DSSCommonConf.java new file mode 100644 index 000000000..161bfb31c --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/conf/DSSCommonConf.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +/** + * @Author alexyang + * @Date 2020/3/27 + */ +public class DSSCommonConf { + + public static final CommonVars DSS_IO_ENV = CommonVars.apply("wds.dss.server.io.env", "BDAP_DEV"); + public static final CommonVars DSS_EXPORT_ENV = CommonVars.apply("wds.dss.server.export.env", "生产中心"); + + public static final CommonVars DSS_EXPORT_URL = CommonVars.apply("wds.dss.server.export.url", "/appcom/tmp/dss"); + public static final CommonVars DSS_INPUT_TOKEN = CommonVars.apply("wds.dss.server.input.token", "QML-AUTH"); + + public static final CommonVars DSS_UPLOAD_PATH = CommonVars.apply("wds.dss.file.upload.dir", "/appcom/tmp/uploads"); + public static final CommonVars DSS_INPUT_RESTFUL_URL = CommonVars.apply("wds.dss.server.input.url", "http://localhost:9004/api/rest_j/v1/dss/input"); + + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/DSSWorkspace.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/DSSWorkspace.java new file mode 100644 index 000000000..c5d85d61d --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/DSSWorkspace.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity; + +/** + * Created by enjoyyin on 2021/7/1. + */ +public interface DSSWorkspace { + + String getWorkspaceName(); + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/IOEnv.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/IOEnv.java new file mode 100644 index 000000000..a4f8e248a --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/IOEnv.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity; + +public enum IOEnv { + /** + * + */ + BDP_DEV,BDP_PRODUCTION,BDAP_DEV,BDAP_PRODUCTION +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/IOType.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/IOType.java new file mode 100644 index 000000000..5d7f23536 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/IOType.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity; + +public enum IOType { + /** + * 区分工程导出和工作流导出 + * basepath/IO.properties + * type=PROJECT 或者 type=FLOE + */ + PROJECT,FLOW,ORCHESTRATOR +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/InputRelation.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/InputRelation.java new file mode 100644 index 000000000..a6a02a491 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/InputRelation.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity; + +public class InputRelation { + + private Long id; + private String type; + private IOEnv sourceEnv; + private Long sourceID; + private IOEnv targetEnv; + private Long targetID; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public IOEnv getSourceEnv() { + return sourceEnv; + } + + public void setSourceEnv(IOEnv sourceEnv) { + this.sourceEnv = sourceEnv; + } + + public Long getSourceID() { + return sourceID; + } + + public void setSourceID(Long sourceID) { + this.sourceID = sourceID; + } + + public IOEnv getTargetEnv() { + return targetEnv; + } + + public void setTargetEnv(IOEnv targetEnv) { + this.targetEnv = targetEnv; + } + + public Long getTargetID() { + return targetID; + } + + public void setTargetID(Long targetID) { + this.targetID = targetID; + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/PublishType.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/PublishType.java new file mode 100644 index 000000000..cdf14a1ab --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/PublishType.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity; + +/** + * @author allenlliu + * @date 2020/12/16 11:53 + */ +public enum PublishType { + FULL,DELTA +} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/Resource.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/Resource.java similarity index 92% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/Resource.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/Resource.java index d4066e401..0dbc3d09d 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/Resource.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/Resource.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,9 +18,6 @@ import java.io.Serializable; -/** - * Created by enjoyyin on 2019/9/24. - */ public class Resource implements Serializable { private String fileName; private String resourceId; diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdge.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdge.java similarity index 88% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdge.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdge.java index c4b30593a..1d64eb759 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdge.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdge.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,9 +16,6 @@ package com.webank.wedatasphere.dss.common.entity.node; -/** - * Created by enjoyyin on 2019/5/14. - */ public interface DSSEdge { String getSource(); diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdgeDefault.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdgeDefault.java similarity index 92% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdgeDefault.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdgeDefault.java index 3e6bc8b13..c6c638916 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdgeDefault.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSEdgeDefault.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,9 +16,6 @@ package com.webank.wedatasphere.dss.common.entity.node; -/** - * Created by enjoyyin on 2019/5/14. - */ public class DSSEdgeDefault implements DSSEdge { private String source; private String target; @@ -68,7 +64,7 @@ public void setTargetLocation(String targetLocation) { @Override public String toString() { - return "DSSEdge{" + + return "DWSEdge{" + "source='" + source + '\'' + ", target='" + target + '\'' + ", sourceLocation='" + sourceLocation + '\'' + diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNode.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNode.java similarity index 90% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNode.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNode.java index 8f6575c83..c37438e53 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNode.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNode.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -22,9 +21,6 @@ import java.util.List; import java.util.Map; -/** - * Created by enjoyyin on 2019/5/14. - */ public interface DSSNode extends Node { Layout getLayout(); diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNodeDefault.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNodeDefault.java similarity index 96% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNodeDefault.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNodeDefault.java index 2d416dfb0..72d24fefa 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNodeDefault.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/DSSNodeDefault.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -23,9 +22,6 @@ import java.util.List; import java.util.Map; -/** - * Created by enjoyyin on 2019/5/14. - */ public class DSSNodeDefault implements DSSNode { private Layout layout; private String id; diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Layout.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Layout.java similarity index 92% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Layout.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Layout.java index d9c4d3b14..3d70f93ef 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Layout.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Layout.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -18,7 +17,7 @@ package com.webank.wedatasphere.dss.common.entity.node; /** - * Created by enjoyyin on 2019/11/12. + * Created by v_wbjftang on 2019/11/12. */ public class Layout { private float x; diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Node.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Node.java similarity index 89% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Node.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Node.java index a802d7515..c3379329b 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Node.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/node/Node.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -20,8 +19,8 @@ import java.util.List; -/** - * Created by enjoyyin on 2019/5/14. +/* + Node 接口 */ public interface Node { String getId(); diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProject.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProject.java new file mode 100644 index 000000000..1beb71528 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/DSSProject.java @@ -0,0 +1,193 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity.project; + +import java.util.Date; +import java.util.List; + +public class DSSProject implements Project { + + private Long id; + private String name; + private String description; + private String createBy; + + private String username; + private Date createTime; + private Date updateTime; + + private String product; + private Integer applicationArea; + private String business; + private Integer workspaceId; + private String workspaceName; + + //stitched + private List viewUsers; + private List editUsers; + private List executeUsers; + + + @Override + public Long getId() { + return id; + } + + @Override + public void setId(Long id) { + this.id = id; + } + + @Override + public String getName() { + return name; + } + + @Override + public void setName(String name) { + this.name = name; + } + + @Override + public String getDescription() { + return description; + } + + @Override + public void setDescription(String description) { + this.description = description; + } + + @Override + public String getCreateBy() { + return createBy; + } + + @Override + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public Integer getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(Integer applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public List getViewUsers() { + return viewUsers; + } + + public void setViewUsers(List viewUsers) { + this.viewUsers = viewUsers; + } + + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + public List getExecuteUsers() { + return executeUsers; + } + + public void setExecuteUsers(List executeUsers) { + this.executeUsers = executeUsers; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + @Override + public String toString() { + return "DSSProject{" + + "id=" + id + + ", name='" + name + '\'' + + ", description='" + description + '\'' + + ", createBy='" + createBy + '\'' + + ", username='" + username + '\'' + + ", createTime=" + createTime + + ", updateTime=" + updateTime + + ", product='" + product + '\'' + + ", applicationArea=" + applicationArea + + ", business='" + business + '\'' + + ", workspaceId=" + workspaceId + + ", workspaceName='" + workspaceName + '\'' + + ", viewUsers=" + viewUsers + + ", editUsers=" + editUsers + + ", executeUsers=" + executeUsers + + '}'; + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/Project.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/Project.java new file mode 100644 index 000000000..4043ce8a8 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/project/Project.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.entity.project; + +/** + * Created by v_wbjftang on 2019/9/16. + */ +public interface Project { + Long getId(); + + void setId(Long id); + + String getName(); + + void setName(String name); + + String getDescription(); + + void setDescription(String description); + + String getCreateBy(); + + void setCreateBy(String createBy); + + +} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSErrorException.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSErrorException.java similarity index 89% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSErrorException.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSErrorException.java index 2fa78edaf..9eeafada5 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSErrorException.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSErrorException.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -20,7 +19,7 @@ import com.webank.wedatasphere.linkis.common.exception.ErrorException; /** - * Created by enjoyyin on 2019/11/6. + * Created by v_wbjftang on 2019/11/6. */ public class DSSErrorException extends ErrorException { diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSRuntimeException.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSRuntimeException.java new file mode 100644 index 000000000..f3b82edfa --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/DSSRuntimeException.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.exception; + +import com.webank.wedatasphere.linkis.common.exception.WarnException; + +/** + * Created by v_wbjftang on 2019/9/24. + */ +public class DSSRuntimeException extends WarnException { + + public DSSRuntimeException(String msg){ + super(100000, msg); + } + + public DSSRuntimeException(int errorCode, String msg) { + super(errorCode, msg); + } + + public DSSRuntimeException(int errorCode, String msg, Throwable e) { + super(errorCode, msg); + initCause(e); + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ErrorCode.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ErrorCode.java new file mode 100644 index 000000000..1e20b247c --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ErrorCode.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.exception; + +/** + * @Author alexyang + * @Date 2020/3/20 + */ +public class ErrorCode { + + public static final int INVALID_PARAMS = 71001; + + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingConsumer.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingConsumer.java new file mode 100644 index 000000000..9c69b1c13 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingConsumer.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.exception; + +/** + * Created by v_wbjftang on 2019/9/20. + */ +@FunctionalInterface +public interface ThrowingConsumer { + void accept(T t) throws E; +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingFunction.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingFunction.java new file mode 100644 index 000000000..a89056aa6 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/exception/ThrowingFunction.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.exception; + +/** + * Created by v_wbjftang on 2019/11/13. + */ +@FunctionalInterface +public interface ThrowingFunction { + R accept(T t) throws E; +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/DSSLabel.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/DSSLabel.java new file mode 100644 index 000000000..0a5eee9f2 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/DSSLabel.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.label; + +import com.webank.wedatasphere.linkis.manager.label.entity.GenericLabel; + +/** + * The basic label class of DSS. All other labels of DSS must extends this class. + */ +public class DSSLabel extends GenericLabel { +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/DSSLabelUtil.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/DSSLabelUtil.java new file mode 100644 index 000000000..27cb5900f --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/DSSLabelUtil.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.label; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * 生成DSSLabel工具类 + */ +public class DSSLabelUtil { + /** + * 临时构建标准labe map + */ + private static Map consltructLabelMap(String label) { + HashMap labelMap = new HashMap<>(); + labelMap.put(LabelKeyConvertor.ROUTE_LABEL_KEY, label); + return labelMap; + } + + /** + * 生成DSSLabel list + * @param label + * @return + */ + public static List createLabelList(String label){ + Map labelParam = consltructLabelMap(label); + LabelInstanceFactory labelInstanceFactory = new LabelInstanceFactory(labelParam); + List dssLabelList = labelInstanceFactory.getLabelList(); + return dssLabelList; + } + + /** + * 生成单一 DSSLabel + * @param label + * @return + */ + public static EnvDSSLabel createLabel(String label){ + Map labelParam = consltructLabelMap(label); + LabelInstanceFactory labelInstanceFactory = new LabelInstanceFactory(labelParam); + EnvDSSLabel envDSSLabel = labelInstanceFactory.getEnvDssLabel(); + return envDSSLabel; + } + + public static void main(String[] args) { + List labelList = DSSLabelUtil.createLabelList("dev"); + System.out.println(labelList.size()); + } + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/EnvDSSLabel.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/EnvDSSLabel.java new file mode 100644 index 000000000..1595101ea --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/EnvDSSLabel.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.label; + +import com.webank.wedatasphere.linkis.manager.label.builder.factory.LabelBuilderFactoryContext; +import com.webank.wedatasphere.linkis.manager.label.entity.Feature; +import com.webank.wedatasphere.linkis.manager.label.entity.annon.ValueSerialNum; +import java.util.HashMap; +import java.util.Map; + +/** + * Used to mark the env of this micro-services. + */ +public class EnvDSSLabel extends DSSLabel { + + public static final String DSS_ENV_LABEL_KEY = "DSSEnv"; + + public EnvDSSLabel(String env) { + setLabelKey(DSS_ENV_LABEL_KEY); + setEnv(env); + } + + public EnvDSSLabel() { + setLabelKey(DSS_ENV_LABEL_KEY); + } + + @Override + public Feature getFeature() { + return Feature.CORE; + } + + public String getEnv() { + if (null == getValue()) { + return null; + } + return getValue().get(DSS_ENV_LABEL_KEY); + } + + @ValueSerialNum(0) + public void setEnv(String env) { + if (null == getValue()) { + setValue(new HashMap<>(1)); + } + getValue().put(DSS_ENV_LABEL_KEY, env); + } + + public static void main(String[] args) { + Map map = new HashMap<>(); + map.put(DSS_ENV_LABEL_KEY, "dev"); + EnvDSSLabel label = (EnvDSSLabel) LabelBuilderFactoryContext.getLabelBuilderFactory().getLabels(map).get(0); + System.out.println("label: " + label.getEnv()); + } + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/LabelInstanceFactory.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/LabelInstanceFactory.java new file mode 100644 index 000000000..3c0ddbe65 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/LabelInstanceFactory.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.label; + +import com.webank.wedatasphere.linkis.manager.label.entity.Label; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.*; +import java.util.stream.Collectors; + +/** + * @author: jinyangrao on 2021/7/16 + * @description: + */ +public class LabelInstanceFactory { + + private static final Logger log = LoggerFactory.getLogger(LabelInstanceFactory.class); + + private Map> labels = null; + + public LabelInstanceFactory(Map labelParam) { + labels = getLabels(labelParam); + } + + private Map> getLabels(Map originalLabels) { + Map originalLabelsMap = LabelKeyConvertor.labelKeyConvert(originalLabels); + return LabelBuilder.buildLabel(originalLabelsMap); + } + + public Map getLabelsMap() { + HashMap resultLabels = new HashMap<>(); + labels.entrySet().stream().map(item -> resultLabels.put(item.getKey(), (DSSLabel) item.getValue())); + return resultLabels; + } + + public List getLabelList() { + List dssLabelList = new ArrayList<>(); + labels.entrySet().stream().map(item -> dssLabelList.add((DSSLabel) (item.getValue()))); + return dssLabelList; + } + + public EnvDSSLabel getEnvDssLabel() { + return (EnvDSSLabel) labels.get(EnvDSSLabel.DSS_ENV_LABEL_KEY); + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/LabelKeyConvertor.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/LabelKeyConvertor.java new file mode 100644 index 000000000..e647a45d1 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/label/LabelKeyConvertor.java @@ -0,0 +1,55 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.label; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.HashMap; +import java.util.Map; + +/** + * @author: jinyangrao on 2021/7/16 + * @description: + */ +public class LabelKeyConvertor { + + private static final Logger log = LoggerFactory.getLogger(LabelKeyConvertor.class); + + + public static String ROUTE_LABEL_KEY = "route"; + + private static final Map convertedlabels = new HashMap<>(); + + static { + convertedlabels.put(ROUTE_LABEL_KEY, EnvDSSLabel.DSS_ENV_LABEL_KEY); + } + + private static String getRealLabelKey(String originalLabelKey) { + return convertedlabels.get(originalLabelKey); + } + + public static Map labelKeyConvert(Map labels) { + Map finalLabels = new HashMap<>(); + for (Map.Entry item : labels.entrySet()) { + String realLabelKey = LabelKeyConvertor.getRealLabelKey(item.getKey()); + finalLabels.put(realLabelKey, item.getValue()); + } + log.info("Get labels map: {}", finalLabels); + return finalLabels; + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/JobStatus.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/JobStatus.java new file mode 100644 index 000000000..9724233e8 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/JobStatus.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol; + +/** + * created by cooperyang on 2020/11/17 + * Description: + */ +public enum JobStatus { + + /** + * JobStatus是用来进行发布任务的状态的追踪 + */ + + Inited("Inited", 1), + Running("Running", 2), + Success("Success", 3), + Failed("Failed", 4); + + private String status; + private int index; + + private JobStatus(String status, int index){ + this.status = status; + this.index = index; + } + + public static boolean isSuccess(String status){ + return Success.status.equals(status); + } + + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectInfoRequest.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectInfoRequest.java new file mode 100644 index 000000000..03a2a8170 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectInfoRequest.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol.project; + +/** + * Created by enjoyyin on 2021/7/2. + */ +public class ProjectInfoRequest { + + private long projectId; + + public long getProjectId() { + return projectId; + } + + public void setProjectId(long projectId) { + this.projectId = projectId; + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectRelationRequest.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectRelationRequest.java new file mode 100644 index 000000000..71bf0a0d5 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectRelationRequest.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol.project; + + +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; + +/** + * created by cooperyang on 2021/1/18 + * Description: + */ +public class ProjectRelationRequest { + + private Long dssProjectId; + + private String appConnName; + + private List dssLabels; + + public ProjectRelationRequest(Long dssProjectId, String appconnName, List dssLabels) { + this.dssProjectId = dssProjectId; + this.appConnName = appconnName; + this.dssLabels = dssLabels; + } + + public Long getDssProjectId() { + return dssProjectId; + } + + public void setDssProjectId(Long dssProjectId) { + this.dssProjectId = dssProjectId; + } + + public String getAppconnName() { + return appConnName; + } + + public void setAppconnName(String appconnName) { + this.appConnName = appconnName; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectRelationResponse.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectRelationResponse.java new file mode 100644 index 000000000..b1072e54b --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/protocol/project/ProjectRelationResponse.java @@ -0,0 +1,76 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol.project; + + +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; + +/** + * created by cooperyang on 2021/1/18 + * Description: + */ +public class ProjectRelationResponse { + + private Long dssProjectId; + + private String appConnName; + + private List dssLabels; + + private Long appInstanceProjectId; + + public ProjectRelationResponse(Long dssProjectId, String appConnName, List dssLabels, Long appInstanceProjectId) { + this.dssProjectId = dssProjectId; + this.appConnName = appConnName; + this.dssLabels = dssLabels; + this.appInstanceProjectId = appInstanceProjectId; + } + + public Long getDssProjectId() { + return dssProjectId; + } + + public void setDssProjectId(Long dssProjectId) { + this.dssProjectId = dssProjectId; + } + + public String getAppConnName() { + return appConnName; + } + + public void setAppConnName(String appConnName) { + this.appConnName = appConnName; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + public Long getAppInstanceProjectId() { + return appInstanceProjectId; + } + + public void setAppInstanceProjectId(Long appInstanceProjectId) { + this.appInstanceProjectId = appInstanceProjectId; + } +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ClassUtils.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ClassUtils.java new file mode 100644 index 000000000..602613d9c --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ClassUtils.java @@ -0,0 +1,123 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.utils; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import java.util.List; +import java.util.Optional; +import java.util.function.Predicate; +import java.util.stream.Collectors; +import org.apache.commons.lang.exception.ExceptionUtils; +import org.reflections.Reflections; + +/** + * Created by enjoyyin on 2021/6/24. + */ +public class ClassUtils { + + private static final ClassHelper CLASS_HELPER = new ClassHelper() { + @Override + protected Reflections getReflections(Class clazz) { + return com.webank.wedatasphere.linkis.common.utils.ClassUtils.reflections(); + } + }; + + public static T getInstance(Class clazz) throws DSSErrorException { + return CLASS_HELPER.getInstance(clazz); + } + + public static T getInstanceOrWarn(Class clazz) { + return CLASS_HELPER.getInstanceOrWarn(clazz); + } + + public static T getInstanceOrDefault(Class clazz, T defaultValue) { + return CLASS_HELPER.getInstanceOrDefault(clazz, defaultValue); + } + + public static T getInstanceOrDefault(Class clazz, Predicate> filterOp, T defaultValue) { + return CLASS_HELPER.getInstanceOrDefault(clazz, filterOp, defaultValue); + } + + public static List getInstances(Class clazz) { + return CLASS_HELPER.getInstances(clazz); + } + + public static List> getClasses(Class clazz) { + return CLASS_HELPER.getClasses(clazz); + } + + public abstract static class ClassHelper { + + protected abstract Reflections getReflections(Class clazz); + + public T getInstance(Class clazz) throws DSSErrorException { + List> factoryClasses = getReflections(clazz).getSubTypesOf(clazz) + .stream().filter(c -> !com.webank.wedatasphere.linkis.common.utils.ClassUtils.isInterfaceOrAbstract(c)).collect(Collectors.toList()); + if(factoryClasses.isEmpty()) { + DSSExceptionUtils.dealErrorException(60053, "Cannot find instance for " + clazz.getSimpleName(), DSSErrorException.class); + } else if(factoryClasses.size() > 1) { + DSSExceptionUtils.dealErrorException(60053, "Too many instances for " + clazz.getSimpleName() + ", exists: " + factoryClasses, + DSSErrorException.class); + } + T t = null; + try { + t = factoryClasses.get(0).newInstance(); + } catch (Exception e) { + DSSExceptionUtils.dealErrorException(60053, "Instance " + clazz.getSimpleName() + " failed", e, DSSErrorException.class); + } + return t; + } + + public T getInstanceOrWarn(Class clazz) { + try { + return getInstance(clazz); + } catch (DSSErrorException e) { + DSSExceptionUtils.dealWarnException(60053, e.getDesc(), ExceptionUtils.getCause(e), DSSRuntimeException.class); + } + return null; + } + + public T getInstanceOrDefault(Class clazz, T defaultValue) { + Optional optional = getReflections(clazz).getSubTypesOf(clazz) + .stream().filter(c -> !com.webank.wedatasphere.linkis.common.utils.ClassUtils.isInterfaceOrAbstract(c) && + !c.isInstance(defaultValue)).findFirst().map(DSSExceptionUtils.map(Class::newInstance)); + return optional.orElse(defaultValue); + } + + public T getInstanceOrDefault(Class clazz, Predicate> filterOp, T defaultValue) { + Optional optional = getReflections(clazz).getSubTypesOf(clazz) + .stream().filter(c -> !com.webank.wedatasphere.linkis.common.utils.ClassUtils.isInterfaceOrAbstract(c) && + filterOp.test(c)).findFirst().map(DSSExceptionUtils.map(Class::newInstance)); + return optional.orElse(defaultValue); + } + + public List getInstances(Class clazz) { + return getReflections(clazz).getSubTypesOf(clazz) + .stream().filter(c -> !com.webank.wedatasphere.linkis.common.utils.ClassUtils.isInterfaceOrAbstract(c)) + .map(DSSExceptionUtils.map(Class::newInstance)).collect(Collectors.toList()); + } + + public List> getClasses(Class clazz) { + return getReflections(clazz).getSubTypesOf(clazz) + .stream().filter(c -> !com.webank.wedatasphere.linkis.common.utils.ClassUtils.isInterfaceOrAbstract(c)) + .collect(Collectors.toList()); + } + + } + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSCommonUtils.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSCommonUtils.java new file mode 100644 index 000000000..37404486d --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSCommonUtils.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.utils; + +import com.google.gson.Gson; +import com.google.gson.GsonBuilder; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +/** + * @Author alexyang + * @Date 2020/3/16 + */ +public class DSSCommonUtils { + + public static final String FLOW_RESOURCE_NAME = "resources"; + + public static final String FLOW_EDGES_NAME = "edges"; + + public static final String FLOW_PARENT_NAME = "parent"; + + + public static final String NODE_RESOURCE_NAME = "resources"; + + public static final String FLOW_NODE_NAME = "nodes"; + + public static final String FLOW_PROP_NAME = "props"; + + public static final String NODE_PROP_NAME = "params"; + + public static final String NODE_PROP_VARIABLE_NAME = "variable"; + + public static final String NODE_ID_NAME = "id"; + + public static final String NODE_NAME_NAME = "title"; + + public static final String FLOW_CONTEXT_ID_PREFIX = "dss.context.id."; + + public static final Gson COMMON_GSON = new GsonBuilder().disableHtmlEscaping().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").create(); + + public static final String ENV_LABEL_VALUE_DEV = "dev"; + + public static final String DSS_LABELS_KEY = "labels"; + + public static final CommonVars DSS_HOME = CommonVars.apply("DSS_HOME", ""); + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSExceptionUtils.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSExceptionUtils.java new file mode 100644 index 000000000..8cac36054 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/DSSExceptionUtils.java @@ -0,0 +1,122 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.utils; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.common.exception.ThrowingConsumer; +import com.webank.wedatasphere.dss.common.exception.ThrowingFunction; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.common.exception.WarnException; +import java.lang.reflect.Constructor; +import java.util.function.Consumer; +import java.util.function.Function; +import org.apache.commons.lang.exception.ExceptionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Created by v_wbjftang on 2019/9/24. + */ +public class DSSExceptionUtils { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSExceptionUtils.class); + + public static Consumer handling( + ThrowingConsumer throwingConsumer) { + return i -> { + try { + throwingConsumer.accept(i); + } catch (Exception e) { + LOGGER.error("execute failed,reason:",e); + throw new DSSRuntimeException(e.getMessage()); + } + }; + } + + public static Function map( + ThrowingFunction throwingFunction) { + return i -> { + try { + return throwingFunction.accept(i); + } catch (Exception e) { + LOGGER.error("execute failed,reason:",e); + throw new DSSRuntimeException(e.getMessage()); + } + }; + } + + public static R tryAndWarn( + ThrowingFunction throwingFunction) { + Function function = i -> { + try { + return throwingFunction.accept(i); + } catch (Exception e) { + LOGGER.error("execute failed,reason:",e); + throw new DSSRuntimeException(53320, ExceptionUtils.getRootCauseMessage(e)); + } + }; + return function.apply(null); + } + + public static void dealErrorException(int errorCode, String errorDesc, Throwable throwable, + Class clazz) throws T{ + T errorException; + try { + Constructor constructor = clazz.getConstructor(int.class, String.class); + errorException = constructor.newInstance(errorCode, errorDesc); + errorException.setErrCode(errorCode); + errorException.setDesc(errorDesc); + } catch (Exception e) { + throw new DSSRuntimeException(errorCode, errorDesc, throwable); + } + errorException.initCause(throwable); + throw errorException; + } + + + public static void dealErrorException(int errorCode, String errorDesc, + Class clazz) throws T { + T errorException; + try { + Constructor constructor = clazz.getConstructor(int.class, String.class); + errorException = constructor.newInstance(errorCode, errorDesc); + errorException.setErrCode(errorCode); + errorException.setDesc(errorDesc); + } catch (Exception e) { + throw new DSSRuntimeException(errorCode, errorDesc, e); + } + throw errorException; + } + + + public static void dealWarnException(int errorCode, String errorDesc, Throwable throwable, + Class clazz) { + T warnException; + try { + Constructor constructor = clazz.getConstructor(int.class, String.class); + warnException = constructor.newInstance(errorCode, errorDesc); + warnException.setErrCode(errorCode); + warnException.setDesc(errorDesc); + warnException.initCause(throwable); + } catch (Exception e) { + throw new DSSRuntimeException(errorCode, errorDesc, throwable); + } + throw warnException; + } + +} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/FileHelper.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/FileHelper.java similarity index 88% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/FileHelper.java rename to dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/FileHelper.java index ea9e61469..820c99cf0 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/FileHelper.java +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/FileHelper.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -20,7 +19,7 @@ import java.io.File; /** - * created by enjoyyin on 2019/6/14 + * created by cooperyang on 2019/6/14 * Description: FileHelper 是 linux 文件系统的帮助类，检查目录是否存在等 */ public class FileHelper { diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/IoUtils.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/IoUtils.java new file mode 100644 index 000000000..41171a790 --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/IoUtils.java @@ -0,0 +1,129 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.utils; + +import com.webank.wedatasphere.dss.common.conf.DSSCommonConf; +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.common.entity.IOType; +import org.apache.commons.io.FileUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.*; +import java.text.SimpleDateFormat; +import java.util.Arrays; +import java.util.Date; +import java.util.Properties; + +/** + * @author allenlliu + * @version 2.0.0 + * @date 2020/03/11 11:18 AM + */ +public class IoUtils { + + private static Logger logger = LoggerFactory.getLogger(IoUtils.class); + private static final String DATE_FORMAT = "yyyyMMddHHmmssSSS"; + private static final String DEFAULT_IO_FILE_NAME = "IO.properties"; + + public static String generateIOPath(String userName,String projectName,String subDir) { + String baseUrl = DSSCommonConf.DSS_EXPORT_URL.getValue(); + String dataStr = new SimpleDateFormat(DATE_FORMAT).format(new Date()); + return addFileSeparator(baseUrl, dataStr, userName, projectName, subDir); + } + + private static String addFileSeparator(String... str) { + return Arrays.stream(str).reduce((a, b) -> a + File.separator + b).orElse(""); + } + + public static OutputStream generateExportOutputStream(String path) throws IOException { + File file = new File(path); + if (!file.getParentFile().exists()) { + FileUtils.forceMkdir(file.getParentFile()); + } + if (file.exists()) { + logger.warn(String.format("%s is exist,delete it", path)); + file.delete(); + } + file.createNewFile(); + return FileUtils.openOutputStream(file, true); + } + + public static InputStream generateInputInputStream(String path)throws IOException{ + return new FileInputStream(path); + } + + public static void generateIOType(IOType ioType, String basePath) throws IOException { + generateIOProperties("type",ioType.name(),basePath); + } + + public static void generateIOProperties(String key,String value,String basePath) throws IOException { + Properties properties = new Properties(); + properties.setProperty(key,value); + + File file = new File(basePath + File.separator + DEFAULT_IO_FILE_NAME); + if (!file.getParentFile().exists()) { + FileUtils.forceMkdir(file.getParentFile()); + } + if(!file.exists()){ + file.createNewFile(); + } + try(FileOutputStream fileOutputStream = FileUtils.openOutputStream(file, true)) { + properties.store(fileOutputStream,""); + } + + } + + public static String readIOProperties(String key,String basepath) throws IOException { + try(FileInputStream inputStream = new FileInputStream(basepath + File.separator + DEFAULT_IO_FILE_NAME)){ + Properties properties = new Properties(); + properties.load(inputStream); + return properties.get(key).toString(); + } + + } + + public static IOType readIOType(String basepath)throws IOException{ + return IOType.valueOf(readIOProperties("type",basepath)); + } + + public static void generateIOEnv(String basePath) throws IOException { + generateIOProperties("env",getDSSServerEnv().name(),basePath); + } + + public static IOEnv readIOEnv(String basePath) throws IOException { + return IOEnv.valueOf(readIOProperties("env",basePath)); + } + + public static IOEnv getDSSServerEnv(){ + //dssserverEnv 是当前dss服务启动的env环境 + return IOEnv.valueOf(DSSCommonConf.DSS_IO_ENV.getValue()); + } + + public static String addVersion(String version){ + String num = String.valueOf(Integer.valueOf(version.substring(1)) + 1); + int length = num.length(); + return version.substring(0,version.length() - length) + num; + } + + public static String subVersion(String version){ + String num = String.valueOf(Integer.valueOf(version.substring(1)) - 1); + int length = num.length(); + return version.substring(0,version.length() - length) + num; + } + +} diff --git a/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ZipHelper.java b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ZipHelper.java new file mode 100644 index 000000000..326d2e14b --- /dev/null +++ b/dss-commons/dss-common/src/main/java/com/webank/wedatasphere/dss/common/utils/ZipHelper.java @@ -0,0 +1,196 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.utils; + + + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.BufferedReader; +import java.io.File; +import java.io.InputStreamReader; +import java.util.ArrayList; +import java.util.List; + +/** + * created by cooperyang on 2019/6/13 + * Description: + */ +public class ZipHelper { + + private static final Logger logger = LoggerFactory.getLogger(ZipHelper.class); + + private static final String ZIP_CMD = "zip"; + private static final String UN_ZIP_CMD = "unzip"; + private static final String RECURSIVE = "-r"; + private static final String ZIP_TYPE = ".zip"; + + public static String zip(String dirPath) throws DSSErrorException { + return zip(dirPath, true); + } + + /** + * ZipHelper可以将传入的path进行打包 + * @param dirPath 需要打包的project路径,绝对路径 + * @return 打包之后的zip包全路径 + */ + public static String zip(String dirPath, boolean deleteOriginDir)throws DSSErrorException { + if(!FileHelper.checkDirExists(dirPath)){ + logger.error("{} 不存在, 不能创建zip文件", dirPath); + throw new DSSErrorException(90001,dirPath + " does not exist, can not zip"); + } + //先用简单的方法，调用新进程进行压缩 + String[] strArr = dirPath.split(File.separator); + String shortPath = strArr[strArr.length - 1]; + String workPath = dirPath.substring(0, dirPath.length() - shortPath.length() - 1); + List list = new ArrayList<>(); + list.add(ZIP_CMD); + list.add(RECURSIVE); + String zipFilePath = shortPath + ZIP_TYPE; + String longZipFilePath = dirPath + ZIP_TYPE; + list.add(zipFilePath); + list.add(shortPath); + ProcessBuilder processBuilder = new ProcessBuilder(list); + processBuilder.redirectErrorStream(true); + processBuilder.directory(new File(workPath)); + BufferedReader infoReader = null; + BufferedReader errorReader = null; + try{ + Process process = processBuilder.start(); + infoReader = new BufferedReader(new InputStreamReader(process.getInputStream())); + errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream())); + String infoLine = null; + while((infoLine = infoReader.readLine()) != null){ + logger.info("process output: {} ", infoLine); + } + String errorLine = null; + StringBuilder errMsg = new StringBuilder(); + while((errorLine = errorReader.readLine()) != null){ + if (StringUtils.isNotEmpty(errorLine)){ + errMsg.append(errorLine).append("\n"); + } + logger.error("process error: {} ", errorLine); + } + int exitCode = process.waitFor(); + if (exitCode != 0){ + throw new DSSErrorException(90002,errMsg.toString()); + } + }catch(final Exception e){ + logger.error("{} 压缩成 zip 文件失败, reason: ", e); + DSSErrorException exception = new DSSErrorException(90003,dirPath + " to zip file failed"); + exception.initCause(e); + throw exception; + } finally { + //删掉整个目录 + IOUtils.closeQuietly(infoReader); + IOUtils.closeQuietly(errorReader); + if(deleteOriginDir) { + File file = new File(dirPath); + logger.info("生成zip文件{}",longZipFilePath); + logger.info("开始删除目录 {}", dirPath); + if (deleteDir(file)){ + logger.info("结束删除目录 {} 成功", dirPath); + }else{ + logger.info("删除目录 {} 失败", dirPath); + } + } + } + return longZipFilePath; + } + + + + public static String unzip(String dirPath)throws DSSErrorException { + File file = new File(dirPath); + if(!file.exists()){ + logger.error("{} 不存在, 不能解压zip文件", dirPath); + throw new DSSErrorException(90001,dirPath + " does not exist, can not unzip"); + } + //先用简单的方法，调用新进程进行压缩 + String[] strArr = dirPath.split(File.separator); + String shortPath = strArr[strArr.length - 1]; + String workPath = dirPath.substring(0, dirPath.length() - shortPath.length() - 1); + List list = new ArrayList<>(); + list.add(UN_ZIP_CMD); + String longZipFilePath = dirPath.replace(ZIP_TYPE,""); + list.add(shortPath); + ProcessBuilder processBuilder = new ProcessBuilder(list); + processBuilder.redirectErrorStream(true); + processBuilder.directory(new File(workPath)); + BufferedReader infoReader = null; + BufferedReader errorReader = null; + try{ + Process process = processBuilder.start(); + infoReader = new BufferedReader(new InputStreamReader(process.getInputStream())); + errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream())); + String infoLine = null; + while((infoLine = infoReader.readLine()) != null){ + logger.info("process output: {} ", infoLine); + } + String errorLine = null; + StringBuilder errMsg = new StringBuilder(); + while((errorLine = errorReader.readLine()) != null){ + if (StringUtils.isNotEmpty(errorLine)){ + errMsg.append(errorLine).append("\n"); + } + logger.error("process error: {} ", errorLine); + } + int exitCode = process.waitFor(); + if (exitCode != 0){ + throw new DSSErrorException(90007,errMsg.toString()); + } + }catch(final Exception e){ + logger.error("{} 解压缩 zip 文件失败, reason: ", e); + DSSErrorException exception = new DSSErrorException(90009,dirPath + " to zip file failed"); + exception.initCause(e); + throw exception; + } finally { + + logger.info("生成解压目录{}", longZipFilePath); + IOUtils.closeQuietly(infoReader); + IOUtils.closeQuietly(errorReader); + } + return longZipFilePath; + } + + private static boolean deleteDir(File dir) { + if (dir.isDirectory()) { + String[] children = dir.list(); + if (children != null && children.length > 0){ + for (String s : children) { + boolean success = deleteDir(new File(dir, s)); + if (!success) { + return false; + } + } + } + } + // 目录此时为空，可以删除 + return dir.delete(); + } + + public static String zipExportProject(String projectPath) throws DSSErrorException { + if(projectPath.endsWith(File.separator)) { + projectPath = projectPath.substring(0, projectPath.lastIndexOf(File.separator)); + } + return ZipHelper.zip(projectPath); + } +} diff --git a/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/label/LabelBuilder.scala b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/label/LabelBuilder.scala new file mode 100644 index 000000000..e2a1b6762 --- /dev/null +++ b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/label/LabelBuilder.scala @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.label + +import com.webank.wedatasphere.linkis.manager.label.entity.Label + +import scala.collection.JavaConverters._ +import java.util + +import com.webank.wedatasphere.linkis.manager.label.builder.factory.{LabelBuilderFactory, LabelBuilderFactoryContext} + + +/** + * @author: jinyangrao on 2021/7/16 + * @description: + */ +object LabelBuilder { + + def buildLabel(labelMap: util.Map[String, Object]): util.Map[String, Label[_]] = { + val labelKeyValueMap = new util.HashMap[String, Label[_]]() + if (null != labelMap && !labelMap.isEmpty) { + val list: util.List[Label[_]] = labelBuilderFactoryProxyMethod.getLabels(labelMap.asInstanceOf[util.Map[String, AnyRef]]) + if (null != list) { + list.asScala.filter(_ != null).foreach { + label => labelKeyValueMap.put(label.getLabelKey, label) + } + } + } + labelKeyValueMap + } + + private def labelBuilderFactoryProxyMethod: LabelBuilderFactory = { + LabelBuilderFactoryContext.getLabelBuilderFactory + } +} diff --git a/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/DSSProtocol.scala b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/DSSProtocol.scala new file mode 100644 index 000000000..595edc597 --- /dev/null +++ b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/DSSProtocol.scala @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol + +/** + * created by cooperyang on 2020/11/17 + * Description: + */ +trait DSSProtocol { + +} diff --git a/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/DSSProtocolObject.scala b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/DSSProtocolObject.scala new file mode 100644 index 000000000..c197ac1c1 --- /dev/null +++ b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/DSSProtocolObject.scala @@ -0,0 +1,66 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol + +import com.webank.wedatasphere.dss.common.label.DSSLabel + + +/** + * Created by v_wbjftang on 2019/11/8. + */ + + + +case class ResponseCreateOrchestrator(orchestratorId: Long, + orchestratorVersionId: Long) + +case class ResponseExportOrchestrator(resourceId: String, + version: String, + orcVersionId: Long) + +case class ResponseImportOrchestrator(orcId: Long) + +case class RequestUpdateWorkflow(userName: String, + flowID: Long, + flowName: String, + description: String, + uses: String) + +case class RequestDeleteWorkflow(userName: String, flowID: Long) + + +case class RequestExportWorkflow(userName: String, + flowID: Long, + projectId: Long, + projectName: String, + workspaceStr: String, + dssLabelList: java.util.List[DSSLabel] + ) + +case class ResponseExportWorkflow(resourceId: String, version: String, flowID: Long) + +case class RequestQueryWorkFlow(userName: String, rootFlowId: Long) + + + + + + + + + + diff --git a/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/framework/ReleaseOrchestratorRequest.scala b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/framework/ReleaseOrchestratorRequest.scala new file mode 100644 index 000000000..25963299f --- /dev/null +++ b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/protocol/framework/ReleaseOrchestratorRequest.scala @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.protocol.framework + +import com.webank.wedatasphere.dss.common.protocol.DSSProtocol + + +/** + * created by cooperyang on 2020/11/17 + * Description: + * DSS框架RPC所需要用到RPC + */ + + + +trait FrameworkProtocol extends DSSProtocol + + +case class ReleaseOrchestratorRequest(releaseUser:String, orchestratorVersionId:Long, orchestratorId:Long, + dssLabel:String, workspaceName:String, workSpaceStr: String) extends FrameworkProtocol + +case class ReleaseOrchestratorResponse(jobId:Long) extends FrameworkProtocol + + + +case class ReleaseOrchestratorStatusRequest(jobId:Long) extends FrameworkProtocol + +case class ReleaseOrchestratorStatusResponse(status:String, jobId:Long, errorMsg:String) extends FrameworkProtocol + + + + + + diff --git a/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/utils/DSSMainHelper.scala b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/utils/DSSMainHelper.scala new file mode 100644 index 000000000..e6380cdf9 --- /dev/null +++ b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/dss/common/utils/DSSMainHelper.scala @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.common.utils + +import com.webank.wedatasphere.linkis.common.conf.{BDPConfiguration, CommonVars, DSSConfiguration} + +import scala.collection.JavaConversions._ + +/** + * @author allenlliu + * @date 2021/3/9 17:18 + */ +object DSSMainHelper { + +// val DSS_DEFAULT_PROPERTY_FILE_NAME = CommonVars("wds.dss.default.property.file", "dss.properties") +// val DSS_SERVER_DEFAULT_PROPERTY_FILE_NAME = CommonVars("wds.dss.server.default.property.file", "%s.properties") + + def formatPropertyFiles(serviceName: String): Unit = { + sys.props.put("wds.linkis.configuration", "dss.properties") + sys.props.put("wds.linkis.server.conf", serviceName+".properties") + } + + //TODO wait for linkis re-written + @Deprecated + def addExtraPropertyFiles(filePaths: String *): Unit = { + sys.props.put("wds.linkis.server.confs", filePaths.mkString(",")) + } + + def getExtraSpringOptions: Array[String] = { + "--spring.profiles.active=dss" +: DSSConfiguration.getAllProperties.filter { case (k, v) => k != null && k.startsWith("spring.")} + .map{ case (k, v) => { + val realKey=k.substring(7) + s"--$realKey=$v" + } + }.toArray + } + +} diff --git a/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/linkis/common/conf/DSSConfiguration.scala b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/linkis/common/conf/DSSConfiguration.scala new file mode 100644 index 000000000..d3a2acd7e --- /dev/null +++ b/dss-commons/dss-common/src/main/scala/com/webank/wedatasphere/linkis/common/conf/DSSConfiguration.scala @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.common.conf + +import java.util.Properties + +/** + * @author allenlliu + * @date 2021/3/10 17:32 + */ +object DSSConfiguration { + + def getAllProperties: Properties = BDPConfiguration.properties + +} diff --git a/dss-commons/dss-contextservice/pom.xml b/dss-commons/dss-contextservice/pom.xml new file mode 100644 index 000000000..950a9ea5a --- /dev/null +++ b/dss-commons/dss-contextservice/pom.xml @@ -0,0 +1,84 @@ + + + + 4.0.0 + + dss-commons + com.webank.wedatasphere.dss + 1.0.0 + + dss-contextservice + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + + com.webank.wedatasphere.linkis + linkis-cs-client + ${linkis.version} + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-scheduler-appconn + ${dss.version} + + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + + + \ No newline at end of file diff --git a/dss-commons/dss-contextservice/src/main/java/com/webank/wedatasphere/dss/contextservice/service/ContextService.java b/dss-commons/dss-contextservice/src/main/java/com/webank/wedatasphere/dss/contextservice/service/ContextService.java new file mode 100644 index 000000000..c646cea10 --- /dev/null +++ b/dss-commons/dss-contextservice/src/main/java/com/webank/wedatasphere/dss/contextservice/service/ContextService.java @@ -0,0 +1,80 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.contextservice.service; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + +/** + * Linkis ContextService访问接口工具类 + * 提供创建ContextID、存储状态信息等到CS服务的接口 + * @Author alexyang + * @Date 2020-0318 + */ +public interface ContextService { + + /** + * 创建ContextID，返回ContextID序列化后的字符串 + * @param flow + * @param user + * @param version + * @return + */ + String createContextID(String workspace, String projectName, String flow, String version, String user); + + /** + * 检查和创建ContextID，返回创建ID后的jsonFlow + * 检查ContextID信息：如果jsonFlow不包含ContextID信息，则创建新的； + * 如果已经有ContextID，判断传入flowVersion是否相同，不同则创建新的ContextID，并返回新创建的ContextID序列化内容；相同则不创建新ID; + * 新创建的ContextID会更新到flowJson里 + * @param jsonFlow + * @param flowVersion + * @param workspace + * @param project + * @param flow + * @param user + * @param fullCheck - true 检查所有参数； false 只检查jsonFlow中ContextID是否存在，存在则不再比较ContextID中version参数等是否相同，且不创建 + * @return + */ + String checkAndCreateContextID(String jsonFlow, String flowVersion, String workspace, String project, String flow, String user, boolean fullCheck); + + /** + * 解析jsonFlow、DWSProject内容，存储到CS里 + * 解析DWSProject、jsonFlow里面的资源、变量、节点依赖等信息，存储到CS服务 + * @param jsonFlow + * @param parentFlowID + */ + void checkAndSaveContext(String jsonFlow, String parentFlowID) throws DSSErrorException; + + /** + * 检查更新ContextID信息，解析传入的jsonFlow、DWSProject内容，存储到CS里 + * @param jsonFlow 必需 + * @param parentFlowId 必需 + * @param workspace + * @param flowVersion + * @param user + * @throws DSSErrorException + */ + String checkAndInitContext(String jsonFlow, String parentFlowId, String workspace, String projectName, String flowName, String flowVersion, String user) throws DSSErrorException; + + /** + * 解析SchedulerFlow信息，并更新ContextID和存储Context信息，调用checkAndInitContext信息 + * @param schedulerFlow + * @return + * @throws DSSErrorException + */ +// String checkAndInitContext(SchedulerFlow schedulerFlow) throws DSSErrorException; +} diff --git a/dss-commons/dss-contextservice/src/main/java/com/webank/wedatasphere/dss/contextservice/service/impl/ContextServiceImpl.java b/dss-commons/dss-contextservice/src/main/java/com/webank/wedatasphere/dss/contextservice/service/impl/ContextServiceImpl.java new file mode 100644 index 000000000..9fe666bae --- /dev/null +++ b/dss-commons/dss-contextservice/src/main/java/com/webank/wedatasphere/dss/contextservice/service/impl/ContextServiceImpl.java @@ -0,0 +1,416 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.contextservice.service.impl; + +import com.google.gson.*; +import com.webank.wedatasphere.dss.common.conf.DSSCommonConf; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.exception.ErrorCode; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.cs.client.ContextClient; +import com.webank.wedatasphere.linkis.cs.client.builder.ContextClientFactory; +import com.webank.wedatasphere.linkis.cs.client.service.CSWorkService; +import com.webank.wedatasphere.linkis.cs.client.service.CSWorkServiceImpl; +import com.webank.wedatasphere.linkis.cs.client.utils.SerializeHelper; +import com.webank.wedatasphere.linkis.cs.common.entity.enumeration.ContextScope; +import com.webank.wedatasphere.linkis.cs.common.entity.enumeration.ContextType; +import com.webank.wedatasphere.linkis.cs.common.entity.enumeration.WorkType; +import com.webank.wedatasphere.linkis.cs.common.entity.object.CSFlowInfos; +import com.webank.wedatasphere.linkis.cs.common.entity.object.LinkisVariable; +import com.webank.wedatasphere.linkis.cs.common.entity.resource.LinkisBMLResource; +import com.webank.wedatasphere.linkis.cs.common.entity.source.*; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.*; + +/** + * @Author alexyang + * @Date 2020-0318 + */ +public class ContextServiceImpl implements ContextService { + + private static final Logger logger = LoggerFactory.getLogger(ContextServiceImpl.class); + private static ContextClient contextClient = ContextClientFactory.getOrCreateContextClient(); + private static ContextService contextService = null; + + private ContextServiceImpl() {} + + public static ContextService getInstance() { + if (null == contextService) { + synchronized (ContextServiceImpl.class) { + if (null == contextService) { + contextService = new ContextServiceImpl(); + } + } + } + return contextService; + } + + @Override + public String createContextID(String workspace, String project, String flow, String version, String user) { + LinkisHAWorkFlowContextID contextID = new LinkisHAWorkFlowContextID(); + contextID.setWorkSpace(workspace); + contextID.setProject(project); + contextID.setFlow(flow); + contextID.setUser(user); + contextID.setVersion(version); + contextID.setEnv(DSSCommonConf.DSS_IO_ENV.getValue()); + try { + contextClient.createContext(contextID); + return SerializeHelper.serializeContextID(contextID); + } catch (Exception e) { + logger.error("createContextID error. workspace : {}, project : {}, flow : {}, version : {}, user : {}", workspace, project, flow, version, user, e); + } + return null; + } + + @Override + public String checkAndCreateContextID(String jsonFlow, String flowVersion, String workspace, String project, String flow, String user, boolean fullCheck) { + JsonObject flowObject = null; + try { + flowObject = new Gson().fromJson(jsonFlow, JsonObject.class); + if (!flowObject.has(CSCommonUtils.CONTEXT_ID_STR) || + StringUtils.isBlank(flowObject.get(CSCommonUtils.CONTEXT_ID_STR).getAsString())|| + !flowObject.get(CSCommonUtils.CONTEXT_ID_STR).isJsonPrimitive()) { + String contextID = createContextID(workspace, + project, + flow, flowVersion, user); + flowObject.addProperty(CSCommonUtils.CONTEXT_ID_STR, contextID); + } else { + boolean updateContextId = false; + LinkisHAWorkFlowContextID contextID = null; + try { + Object contextIDObj = SerializeHelper.deserializeContextID(flowObject.get(CSCommonUtils.CONTEXT_ID_STR).getAsString()); + contextID = (LinkisHAWorkFlowContextID)contextIDObj; + if (fullCheck) { + if (!DSSCommonConf.DSS_IO_ENV.getValue().equalsIgnoreCase(contextID.getEnv())) { + updateContextId = true; + } else if (StringUtils.isBlank(contextID.getProject()) + || StringUtils.isBlank(contextID.getFlow()) + || StringUtils.isBlank(contextID.getVersion())) { + updateContextId = false; + } else if ((null != contextID.getWorkSpace() && !contextID.getWorkSpace().equalsIgnoreCase(workspace)) + || !contextID.getProject().equalsIgnoreCase(project) + || !contextID.getFlow().equalsIgnoreCase(flow) + || !contextID.getVersion().equalsIgnoreCase(flowVersion)) { + updateContextId = true; + } else { + updateContextId = false; + } + } + } catch (ErrorException e0) { + logger.error("Invalid contextID : {}, please contact with administrator", flowObject.get(CSCommonUtils.CONTEXT_ID_STR)); + } + if (updateContextId) { + String newContextID = createContextID(workspace, + project, + flow, flowVersion, user); + flowObject.addProperty(CSCommonUtils.CONTEXT_ID_STR, newContextID); + logger.info("UpdateContextId true, old contextID : {}, new contextID : {}", CSCommonUtils.gson.toJson(contextID), CSCommonUtils.gson.toJson(newContextID)); + } + } + } catch (Exception e) { + logger.error("Invalid json : {}", jsonFlow, e); + } + if (null != flowObject) { + return flowObject.toString(); + } else { + return jsonFlow; + } + } + + @Override + public void checkAndSaveContext(String jsonFlow,String parentFlowID) throws DSSErrorException{ + logger.info("jsonFlow => \n" + jsonFlow); + try { + JsonObject flowObject = new Gson().fromJson(jsonFlow, JsonObject.class); + if (!flowObject.has(CSCommonUtils.CONTEXT_ID_STR) || !flowObject.get(CSCommonUtils.CONTEXT_ID_STR).isJsonPrimitive()) { + logger.error("Did not have invalid contextID, save context failed."); + return; + } else { + String contextIDStr = flowObject.get(CSCommonUtils.CONTEXT_ID_STR).getAsString(); + // ①reset原有key 这里只清理 + CSWorkService csWorkService = CSWorkServiceImpl.getInstance(); + List workTypes = new ArrayList<>(); + workTypes.add(WorkType.WORKSPACE); + workTypes.add(WorkType.PROJECT); + workTypes.add(WorkType.FLOW); + csWorkService.initContextServiceInfo(contextIDStr, workTypes); + + // ②解析和保存新的 UDF、Resource、Variable + // 保存Workspace和Project的资源参数等 +// if (null != project.getProjectResources() && project.getProjectResources().size() > 0) { +// saveContextResource(contextIDStr, project.getProjectResources(), contextClient, CSCommonUtils.PROJECT_RESOURCE_PREFIX); +// } + + // 保存flow的资源 + if (flowObject.has(DSSCommonUtils.FLOW_RESOURCE_NAME)) { + JsonArray flowRes = flowObject.get(DSSCommonUtils.FLOW_RESOURCE_NAME).getAsJsonArray(); + saveContextResource(contextIDStr, flowRes, contextClient, CSCommonUtils.FLOW_RESOURCE_PREFIX, null); + } + if (flowObject.has(DSSCommonUtils.FLOW_PROP_NAME)) { + JsonElement flowProp = flowObject.get(DSSCommonUtils.FLOW_PROP_NAME); + saveContextVariable(contextIDStr, flowProp, contextClient, CSCommonUtils.FLOW_VARIABLE_PREFIX, null); + } + // todo udf + + // 保存节点的资源 + if (flowObject.has(DSSCommonUtils.FLOW_NODE_NAME)) { + JsonArray nodes = flowObject.get(DSSCommonUtils.FLOW_NODE_NAME).getAsJsonArray(); + for (JsonElement node : nodes) { + JsonObject json = node.getAsJsonObject(); + if (json.has(DSSCommonUtils.NODE_RESOURCE_NAME)) { + JsonArray nodeRes = json.get(DSSCommonUtils.NODE_RESOURCE_NAME).getAsJsonArray(); + saveContextResource(contextIDStr, nodeRes, contextClient, + CSCommonUtils.NODE_PREFIX, json.get(DSSCommonUtils.NODE_NAME_NAME).getAsString()); + } + if (json.has(DSSCommonUtils.NODE_PROP_NAME)) { + JsonObject nodePropObj = json.get(DSSCommonUtils.NODE_PROP_NAME).getAsJsonObject(); + if (nodePropObj.has(DSSCommonUtils.NODE_PROP_VARIABLE_NAME)) { + JsonElement nodeVariables = nodePropObj.get(DSSCommonUtils.NODE_PROP_VARIABLE_NAME); + saveContextVariable(contextIDStr, nodeVariables, contextClient, + CSCommonUtils.NODE_PREFIX, json.get(DSSCommonUtils.NODE_NAME_NAME).getAsString()); + } + } + } + } + // 保存info信息 + saveFlowInfo(contextIDStr, parentFlowID, jsonFlow); + } + } catch (Exception e) { + logger.error("CheckAndSaveContext error. jsonFlow : {}, parentFlowId : {}, e : ", jsonFlow, parentFlowID, e); + throw new DSSErrorException(ErrorCode.INVALID_PARAMS, "CheckAndSaveContext error : " + e.getMessage()); + } + } + + @Override + public String checkAndInitContext(String jsonFlow, String parentFlowId, String workspace, String projectName, String flowName, String flowVersion, String user) throws DSSErrorException { + if (StringUtils.isBlank(jsonFlow) ) { + logger.error("Invalid jsonFlow : {} or parentFlowId : {}.", jsonFlow, parentFlowId); + throw new DSSErrorException(ErrorCode.INVALID_PARAMS, "Invalid jsonFlow : " + jsonFlow + ", or parentFlowId : " + parentFlowId + "."); + } + jsonFlow = checkAndCreateContextID(jsonFlow, flowVersion, workspace, projectName, flowName, user, true); + checkAndSaveContext(jsonFlow,parentFlowId); + JsonObject flowObject = new Gson().fromJson(jsonFlow, JsonObject.class); + return flowObject.get(CSCommonUtils.CONTEXT_ID_STR).getAsString(); + } + + + + private void saveContextVariable(String contextIDStr, JsonElement variables, ContextClient contextClient, String variablePrefix, String nodeName) { + try { + if (variables.isJsonArray()) { + JsonArray flowProp = variables.getAsJsonArray(); + for (JsonElement prop : flowProp) { + Set> entrySet = prop.getAsJsonObject().entrySet(); + // assign entry num is 1 + if (null != entrySet) { + for (Map.Entry entry : entrySet) { + if (!entry.getValue().isJsonPrimitive()) { + continue; + } + saveContextVariableKeyValue(contextIDStr, contextClient, variablePrefix, entry, nodeName); + } + } + + } + } else if (variables.isJsonObject()) { + JsonObject variableJson = variables.getAsJsonObject(); + for (Map.Entry entry : variableJson.entrySet()) { + if (!entry.getValue().isJsonPrimitive()) { + continue; + } + saveContextVariableKeyValue(contextIDStr, contextClient, variablePrefix, entry, nodeName); + } + } else { + logger.error("Invalid JsonElement variables : {}", variables.toString()); + } + + } catch (ErrorException e) { + logger.error("SaveContextVariable failed. contextIDStr : {}, variables : {}, variablePrefix : {}, e : ", + contextIDStr, variables.toString(), variablePrefix, e); + } + + } + + private void saveContextVariableKeyValue(String contextIDStr, ContextClient contextClient, String uniKeyPrefix, Map.Entry entry, String nodeName) throws ErrorException { + String contextKeyPrefix = null; + switch (uniKeyPrefix) { + case CSCommonUtils.WORKSPACE_VARIABLE_PREFIX: + case CSCommonUtils.PROJECT_VARIABLE_PREFIX: + case CSCommonUtils.FLOW_VARIABLE_PREFIX: + contextKeyPrefix = uniKeyPrefix; + break; + case CSCommonUtils.NODE_PREFIX: + contextKeyPrefix = uniKeyPrefix + nodeName + "." + CSCommonUtils.RESOURCE_PREFIX; + break; + default: + logger.error("Invalid contextKeyPrefix : {}", uniKeyPrefix); + return; + } + LinkisVariable linkisVariable = new LinkisVariable(); + linkisVariable.setKey(entry.getKey()); + linkisVariable.setValue(entry.getValue().getAsString()); + ContextKey contextKey = new CommonContextKey(); + contextKey.setKey(contextKeyPrefix + linkisVariable.getKey()); + contextKey.setContextType(ContextType.OBJECT); + contextKey.setContextScope(ContextScope.PUBLIC); + ContextValue contextValue = new CommonContextValue(); + contextValue.setValue(linkisVariable); + ContextKeyValue contextKeyValue = new CommonContextKeyValue(contextKey, contextValue); + contextClient.setContextKeyValue(SerializeHelper.deserializeContextID(contextIDStr), contextKeyValue); + } + + private void saveContextResource(String contextIDStr, JsonArray flowRes, ContextClient contextClient, String uniKeyPrefix, String nodeName) throws ErrorException { + String contextKeyPrefix = null; + switch (uniKeyPrefix) { + case CSCommonUtils.WORKSPACE_RESOURCE_PREFIX: + case CSCommonUtils.PROJECT_RESOURCE_PREFIX: + case CSCommonUtils.FLOW_RESOURCE_PREFIX: + contextKeyPrefix = uniKeyPrefix; + break; + case CSCommonUtils.NODE_PREFIX: + contextKeyPrefix = uniKeyPrefix + nodeName + "." + CSCommonUtils.RESOURCE_PREFIX; + break; + default: + logger.error("Invalid contextKeyPrefix : {}", uniKeyPrefix); + return; + } + for (JsonElement res : flowRes) { + LinkisBMLResource bmlResource = new LinkisBMLResource(); + JsonObject json = res.getAsJsonObject(); + if (!json.has("fileName") || !json.has("resourceId") || !json.has("version")) { + logger.warn("Invalid resource: res : {}, contextidStr : {}, all res : {} ", CSCommonUtils.gson.toJson(json), contextIDStr, CSCommonUtils.gson.toJson(flowRes)); + continue; + } + if (json.get("fileName") instanceof JsonNull || json.get("resourceId") instanceof JsonNull || json.get("version") instanceof JsonNull) { + logger.warn("Invalid resource: res : {}, contextidStr : {}, all res : {} ", CSCommonUtils.gson.toJson(json), contextIDStr, CSCommonUtils.gson.toJson(flowRes)); + continue; + } + bmlResource.setDownloadedFileName(json.get("fileName").getAsString()); + bmlResource.setResourceId(json.get("resourceId").getAsString()); + bmlResource.setVersion(json.get("version").getAsString()); + ContextKey contextKey = new CommonContextKey(); + contextKey.setKey(contextKeyPrefix + bmlResource.getDownloadedFileName()); + contextKey.setContextScope(ContextScope.PUBLIC); + contextKey.setContextType(ContextType.RESOURCE); + ContextValue contextValue = new CommonContextValue(); + contextValue.setValue(bmlResource); + ContextKeyValue contextKeyValue = new CommonContextKeyValue(contextKey, contextValue); + contextClient.setContextKeyValue(SerializeHelper.deserializeContextID(contextIDStr), contextKeyValue); + //// todo test + logger.info("Debug: saved contextKeyValue : {}", CSCommonUtils.gson.toJson(contextKeyValue)); + } + } + + private void saveContextResource(String contextIDStr, List resourceList, ContextClient contextClient, String uniKeyPrefix) throws ErrorException { + + String contextKeyPrefix = null; + switch (uniKeyPrefix) { + case CSCommonUtils.WORKSPACE_RESOURCE_PREFIX: + case CSCommonUtils.PROJECT_RESOURCE_PREFIX: + case CSCommonUtils.FLOW_RESOURCE_PREFIX: + contextKeyPrefix = uniKeyPrefix; + break; + default: + logger.error("Invalid contextKeyPrefix : {}", uniKeyPrefix); + return; + } + for (Resource res : resourceList) { + LinkisBMLResource bmlResource = new LinkisBMLResource(); + bmlResource.setDownloadedFileName(res.getFileName()); + bmlResource.setResourceId(res.getResourceId()); + bmlResource.setVersion(res.getVersion()); + ContextKey contextKey = new CommonContextKey(); + contextKey.setKey(contextKeyPrefix + bmlResource.getDownloadedFileName()); + contextKey.setContextScope(ContextScope.PUBLIC); + contextKey.setContextType(ContextType.RESOURCE); + ContextValue contextValue = new CommonContextValue(); + contextValue.setValue(bmlResource); + ContextKeyValue contextKeyValue = new CommonContextKeyValue(contextKey, contextValue); + contextClient.setContextKeyValue(SerializeHelper.deserializeContextID(contextIDStr), contextKeyValue); + //// todo test + logger.info("Debug: saved contextKeyValue : {}", CSCommonUtils.gson.toJson(contextKeyValue)); + } + } + + + /** + * 保存节点间关系信息 + * @param contextIDStr + * @param parentFlowID + * @param parentFlowID + */ + private void saveFlowInfo(String contextIDStr, String parentFlowID, String flowJson) { + if (StringUtils.isBlank(contextIDStr)) { + return; + } + ContextClient contextClient = ContextClientFactory.getOrCreateContextClient(); + JsonObject flowJsonObject = new Gson().fromJson(flowJson, JsonObject.class); + CSFlowInfos flowInfos = new CSFlowInfos(); + Map flowInfoMap = new HashMap<>(); + // 保存边信息 + JsonArray flowEdges = new JsonArray(); + if (flowJsonObject.has(DSSCommonUtils.FLOW_EDGES_NAME)) { + flowEdges = flowJsonObject.getAsJsonArray(DSSCommonUtils.FLOW_EDGES_NAME); + } + flowInfoMap.put(DSSCommonUtils.FLOW_EDGES_NAME, flowEdges); + // 保存父节点 可以为空 + flowInfoMap.put(DSSCommonUtils.FLOW_PARENT_NAME, parentFlowID); + //保存节点名 + JsonObject idNodeNameJson = new JsonObject(); + if (flowJsonObject.has(DSSCommonUtils.FLOW_NODE_NAME)) { + JsonArray nodes = flowJsonObject.getAsJsonArray(DSSCommonUtils.FLOW_NODE_NAME); + for (JsonElement element : nodes) { + JsonObject jsonObject = element.getAsJsonObject(); + String id = jsonObject.get(DSSCommonUtils.NODE_ID_NAME).getAsString(); + String name = jsonObject.get(DSSCommonUtils.NODE_NAME_NAME).getAsString(); + idNodeNameJson.addProperty(id, name); + } + } + flowInfoMap.put(CSCommonUtils.ID_NODE_NAME, idNodeNameJson); + flowInfos.setInfos(flowInfoMap); + ContextKey contextKey = new CommonContextKey(); + contextKey.setContextType(ContextType.OBJECT); + contextKey.setContextScope(ContextScope.PUBLIC); + contextKey.setKey(CSCommonUtils.FLOW_INFOS); + ContextValue contextValue = new CommonContextValue(); + contextValue.setValue(flowInfos); + ContextKeyValue contextKeyValue = new CommonContextKeyValue(contextKey, contextValue); + try { + contextClient.setContextKeyValue(SerializeHelper.deserializeContextID(contextIDStr), contextKeyValue); + //// todo test + logger.info("ContextID : {}, \nContextKey : {}, \nContextValue : {}", + CSCommonUtils.gson.toJson(SerializeHelper.deserializeContextID(contextIDStr)), + CSCommonUtils.gson.toJson(contextKey), + CSCommonUtils.gson.toJson(contextValue)); + ContextValue contextValue1 = contextClient.getContextValue(SerializeHelper.deserializeContextID(contextIDStr), contextKey); + logger.info(CSCommonUtils.gson.toJson(contextValue1)); + } catch (ErrorException e) { + logger.error("Set ContextKeyValue error. contextIDStr "); + } + } + + + +} diff --git a/dss-commons/dss-sender-service/pom.xml b/dss-commons/dss-sender-service/pom.xml new file mode 100644 index 000000000..41cd35829 --- /dev/null +++ b/dss-commons/dss-sender-service/pom.xml @@ -0,0 +1,57 @@ + + + + + + dss-commons + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-sender-service + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + provided + + + + + \ No newline at end of file diff --git a/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/DSSSenderService.java b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/DSSSenderService.java new file mode 100644 index 000000000..c160f3c27 --- /dev/null +++ b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/DSSSenderService.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.sender.service; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.util.List; + +/** + * @author allenlliu + * @date 2021/6/24 17:09 + */ +public interface DSSSenderService { + + Sender getOrcSender(); + + Sender getOrcSender(List dssLabels); + + Sender getScheduleOrcSender(); + + Sender getWorkflowSender(List dssLabels); + + Sender getWorkflowSender(); + + Sender getSchedulerWorkflowSender(); + + Sender getProjectServerSender(); + +} + diff --git a/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/DSSSenderServiceFactory.java b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/DSSSenderServiceFactory.java new file mode 100644 index 000000000..5d36f92ed --- /dev/null +++ b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/DSSSenderServiceFactory.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.sender.service; + +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.sender.service.impl.DSSSenderServiceImpl; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * @author allenlliu + * @date 2021/6/28 11:41 + */ +public class DSSSenderServiceFactory { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSSenderServiceFactory.class); + private static final DSSSenderService service; + + static { + service = ClassUtils.getInstanceOrDefault(DSSSenderService.class, new DSSSenderServiceImpl()); + LOGGER.info("Use {} to instance a available DSSSenderService.", service.getClass().getName()); + } + + + public static DSSSenderService getOrCreateServiceInstance() { + return service; + } + + +} diff --git a/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/conf/DSSSenderServiceConf.java b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/conf/DSSSenderServiceConf.java new file mode 100644 index 000000000..41539abe6 --- /dev/null +++ b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/conf/DSSSenderServiceConf.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.sender.service.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + +/** + * @author allenlliu + * @date 2021/6/24 17:23 + */ +public class DSSSenderServiceConf { + public static final CommonVars ORCHESTRATOR_SERVER_DEV_NAME = + CommonVars.apply("wds.dss.orc.server.dev.name", "DSS-Framework-Orchestrator-Server-Dev"); + + public static final CommonVars DSS_WORKFLOW_APPLICATION_NAME_DEV = + CommonVars.apply("wds.dss.workflow.name.dev", "dss-workflow-server-dev"); + + public static final CommonVars PROJECT_SERVER_NAME = + CommonVars.apply("wds.dss.project.sever.name", "dss-framework-project-server"); + +} diff --git a/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/impl/DSSSenderServiceImpl.java b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/impl/DSSSenderServiceImpl.java new file mode 100644 index 000000000..93ca057bc --- /dev/null +++ b/dss-commons/dss-sender-service/src/main/java/com/webank/wedatasphere/dss/sender/service/impl/DSSSenderServiceImpl.java @@ -0,0 +1,71 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.sender.service.impl; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.sender.service.DSSSenderService; +import com.webank.wedatasphere.dss.sender.service.conf.DSSSenderServiceConf; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.util.List; + +/** + * @author allenlliu + * @date 2021/6/24 17:13 + */ +public class DSSSenderServiceImpl implements DSSSenderService { + + private final Sender orcSender = Sender.getSender(DSSSenderServiceConf.ORCHESTRATOR_SERVER_DEV_NAME.getValue()); + + private final Sender workflowSender = Sender.getSender(DSSSenderServiceConf.DSS_WORKFLOW_APPLICATION_NAME_DEV.getValue()); + + private final Sender projectSender = Sender.getSender(DSSSenderServiceConf.PROJECT_SERVER_NAME.getValue()); + @Override + public Sender getOrcSender() { + return orcSender; + } + + @Override + public Sender getOrcSender(List dssLabels) { + return orcSender; + } + + @Override + public Sender getScheduleOrcSender() { + return orcSender; + } + + @Override + public Sender getWorkflowSender(List dssLabels) { + return workflowSender; + } + + @Override + public Sender getWorkflowSender() { + return workflowSender; + } + + @Override + public Sender getSchedulerWorkflowSender() { + return workflowSender; + } + + @Override + public Sender getProjectServerSender() { + return projectSender; + } + +} diff --git a/dss-commons/pom.xml b/dss-commons/pom.xml new file mode 100644 index 000000000..be492b6f4 --- /dev/null +++ b/dss-commons/pom.xml @@ -0,0 +1,37 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-commons + pom + + + dss-common + dss-contextservice + dss-sender-service + + + \ No newline at end of file diff --git a/dss-flow-execution-entrance/bin/start-dss-flow-execution-entrance.sh b/dss-flow-execution-entrance/bin/start-dss-flow-execution-entrance.sh deleted file mode 100644 index 9bd4a006c..000000000 --- a/dss-flow-execution-entrance/bin/start-dss-flow-execution-entrance.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/bin/bash -cd `dirname $0` -cd .. -HOME=`pwd` - -export SERVER_PID=$HOME/bin/linkis.pid -export SERVER_LOG_PATH=$HOME/logs -export SERVER_CLASS=com.webank.wedatasphere.linkis.DataWorkCloudApplication - -if test -z "$SERVER_HEAP_SIZE" -then - export SERVER_HEAP_SIZE="512M" -fi - -if test -z "$SERVER_JAVA_OPTS" -then - export SERVER_JAVA_OPTS=" -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$HOME/logs/linkis-gc.log" -fi - -if [[ -f "${SERVER_PID}" ]]; then - pid=$(cat ${SERVER_PID}) - if kill -0 ${pid} >/dev/null 2>&1; then - echo "Server is already running." - exit 1 - fi -fi - -nohup java $SERVER_JAVA_OPTS -cp $HOME/conf:$HOME/lib/* $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/linkis.out & -pid=$! -if [[ -z "${pid}" ]]; then - echo "server $SERVER_NAME start failed!" - exit 1 -else - echo "server $SERVER_NAME start succeeded!" - echo $pid > $SERVER_PID - sleep 1 -fi - - - - - - - - - - - - diff --git a/dss-flow-execution-entrance/bin/stop-dss-flow-execution-entrance.sh b/dss-flow-execution-entrance/bin/stop-dss-flow-execution-entrance.sh deleted file mode 100644 index ce9cb2ee2..000000000 --- a/dss-flow-execution-entrance/bin/stop-dss-flow-execution-entrance.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -cd `dirname $0` -cd .. -HOME=`pwd` - -export FLOW_ENTRANCE_HOME_PID=$HOME/bin/linkis.pid - -function wait_for_FLOW_ENTRANCE_HOME_to_die() { - local pid - local count - pid=$1 - timeout=$2 - count=0 - timeoutTime=$(date "+%s") - let "timeoutTime+=$timeout" - currentTime=$(date "+%s") - forceKill=1 - - while [[ $currentTime -lt $timeoutTime ]]; do - $(kill ${pid} > /dev/null 2> /dev/null) - if kill -0 ${pid} > /dev/null 2>&1; then - sleep 3 - else - forceKill=0 - break - fi - currentTime=$(date "+%s") - done - - if [[ forceKill -ne 0 ]]; then - $(kill -9 ${pid} > /dev/null 2> /dev/null) - fi -} - -if [[ ! -f "${FLOW_ENTRANCE_HOME_PID}" ]]; then - echo "FlowExecution Entrance is not running" -else - pid=$(cat ${FLOW_ENTRANCE_HOME_PID}) - if [[ -z "${pid}" ]]; then - echo "FlowExecution Entrance is not running" - else - wait_for_FLOW_ENTRANCE_HOME_to_die $pid 40 - $(rm -f ${FLOW_ENTRANCE_HOME_PID}) - echo "FlowExecution Entrance is stopped." - fi -fi diff --git a/dss-flow-execution-entrance/pom.xml b/dss-flow-execution-entrance/pom.xml deleted file mode 100644 index 669ca30cd..000000000 --- a/dss-flow-execution-entrance/pom.xml +++ /dev/null @@ -1,126 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-flow-execution-entrance - - - - com.webank.wedatasphere.linkis - linkis-ujes-entrance - ${linkis.version} - - - org.apache.poi - ooxml-schemas - - - - - com.webank.wedatasphere.linkis - linkis-cloudRPC - ${linkis.version} - - - com.webank.wedatasphere.dss - dss-linkis-node-execution - ${dss.version} - - - com.ibm.icu - icu4j - - - - - - com.webank.wedatasphere.dss - dss-scheduler-appjoint-core - ${dss.version} - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - dss-flow-execution-entrance - false - false - - src/main/assembly/distribution.xml - - - - - - - ${basedir}/src/main/resources - - **/*.properties - **/*.xml - **/*.yml - - - - ${project.artifactId}-${project.version} - - - \ No newline at end of file diff --git a/dss-flow-execution-entrance/src/main/assembly/distribution.xml b/dss-flow-execution-entrance/src/main/assembly/distribution.xml deleted file mode 100644 index bb09aad22..000000000 --- a/dss-flow-execution-entrance/src/main/assembly/distribution.xml +++ /dev/null @@ -1,303 +0,0 @@ - - - - dss-flow-execution-entrance - - zip - - true - dss-flow-execution-entrance - - - - - - lib - true - true - false - true - true - - - antlr:antlr:jar - aopalliance:aopalliance:jar - asm:asm:jar - cglib:cglib:jar - - - - - - - com.fasterxml.jackson.core:jackson-annotations:jar - com.fasterxml.jackson.core:jackson-core:jar - com.fasterxml.jackson.core:jackson-databind:jar - - com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar - com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar - com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar - com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar - com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar - com.fasterxml.jackson.module:jackson-module-parameter-names:jar - com.fasterxml.jackson.module:jackson-module-paranamer:jar - com.fasterxml.jackson.module:jackson-module-scala_2.11:jar - - - com.google.code.findbugs:annotations:jar - - com.google.code.gson:gson:jar - com.google.guava:guava:jar - com.google.inject:guice:jar - com.netflix.archaius:archaius-core:jar - com.netflix.eureka:eureka-client:jar - com.netflix.eureka:eureka-core:jar - com.netflix.hystrix:hystrix-core:jar - com.netflix.netflix-commons:netflix-commons-util:jar - com.netflix.netflix-commons:netflix-eventbus:jar - com.netflix.netflix-commons:netflix-infix:jar - com.netflix.netflix-commons:netflix-statistics:jar - com.netflix.ribbon:ribbon:jar - com.netflix.ribbon:ribbon-core:jar - com.netflix.ribbon:ribbon-eureka:jar - com.netflix.ribbon:ribbon-httpclient:jar - com.netflix.ribbon:ribbon-loadbalancer:jar - com.netflix.ribbon:ribbon-transport:jar - com.netflix.servo:servo-core:jar - com.sun.jersey.contribs:jersey-apache-client4:jar - com.sun.jersey:jersey-client:jar - com.sun.jersey:jersey-core:jar - com.sun.jersey:jersey-json:jar - com.sun.jersey:jersey-server:jar - com.sun.jersey:jersey-servlet:jar - com.sun.xml.bind:jaxb-impl:jar - com.thoughtworks.paranamer:paranamer:jar - com.thoughtworks.xstream:xstream:jar - com.webank.wedatasphere.linkis:linkis-common:jar - com.webank.wedatasphere.linkis:linkis-module:jar - commons-beanutils:commons-beanutils:jar - commons-codec:commons-codec:jar - commons-collections:commons-collections:jar - commons-configuration:commons-configuration:jar - commons-dbcp:commons-dbcp:jar - commons-io:commons-io:jar - commons-jxpath:commons-jxpath:jar - commons-lang:commons-lang:jar - commons-logging:commons-logging:jar - commons-net:commons-net:jar - commons-pool:commons-pool:jar - io.micrometer:micrometer-core:jar - io.netty:netty-all:jar - io.netty:netty-buffer:jar - io.netty:netty-codec:jar - io.netty:netty-codec-http:jar - io.netty:netty-common:jar - io.netty:netty-handler:jar - io.netty:netty-transport:jar - io.netty:netty-transport-native-epoll:jar - io.reactivex:rxjava:jar - io.reactivex:rxnetty:jar - io.reactivex:rxnetty-contexts:jar - io.reactivex:rxnetty-servo:jar - javax.activation:activation:jar - javax.annotation:javax.annotation-api:jar - javax.inject:javax.inject:jar - javax.servlet:javax.servlet-api:jar - javax.validation:validation-api:jar - javax.websocket:javax.websocket-api:jar - javax.ws.rs:javax.ws.rs-api:jar - javax.xml.bind:jaxb-api:jar - javax.xml.stream:stax-api:jar - joda-time:joda-time:jar - mysql:mysql-connector-java:jar - org.antlr:antlr-runtime:jar - org.antlr:stringtemplate:jar - org.apache.commons:commons-math:jar - org.apache.httpcomponents:httpclient:jar - org.apache.logging.log4j:log4j-api:jar - org.apache.logging.log4j:log4j-core:jar - org.apache.logging.log4j:log4j-jul:jar - org.apache.logging.log4j:log4j-slf4j-impl:jar - org.aspectj:aspectjweaver:jar - org.bouncycastle:bcpkix-jdk15on:jar - org.bouncycastle:bcprov-jdk15on:jar - org.codehaus.jackson:jackson-jaxrs:jar - org.codehaus.jackson:jackson-xc:jar - org.codehaus.jettison:jettison:jar - org.codehaus.woodstox:stax2-api:jar - org.codehaus.woodstox:woodstox-core-asl:jar - org.eclipse.jetty:jetty-annotations:jar - org.eclipse.jetty:jetty-client:jar - org.eclipse.jetty:jetty-continuation:jar - org.eclipse.jetty:jetty-http:jar - org.eclipse.jetty:jetty-io:jar - org.eclipse.jetty:jetty-plus:jar - org.eclipse.jetty:jetty-security:jar - org.eclipse.jetty:jetty-server:jar - org.eclipse.jetty:jetty-servlet:jar - org.eclipse.jetty:jetty-servlets:jar - org.eclipse.jetty:jetty-util:jar - org.eclipse.jetty:jetty-webapp:jar - org.eclipse.jetty:jetty-xml:jar - org.eclipse.jetty.websocket:javax-websocket-client-impl:jar - org.eclipse.jetty.websocket:javax-websocket-server-impl:jar - org.eclipse.jetty.websocket:websocket-api:jar - org.eclipse.jetty.websocket:websocket-client:jar - org.eclipse.jetty.websocket:websocket-common:jar - org.eclipse.jetty.websocket:websocket-server:jar - org.eclipse.jetty.websocket:websocket-servlet:jar - org.glassfish.hk2:class-model:jar - org.glassfish.hk2:config-types:jar - org.glassfish.hk2.external:aopalliance-repackaged:jar - org.glassfish.hk2.external:asm-all-repackaged:jar - org.glassfish.hk2.external:bean-validator:jar - org.glassfish.hk2.external:javax.inject:jar - org.glassfish.hk2:hk2:jar - org.glassfish.hk2:hk2-api:jar - org.glassfish.hk2:hk2-config:jar - org.glassfish.hk2:hk2-core:jar - org.glassfish.hk2:hk2-locator:jar - org.glassfish.hk2:hk2-runlevel:jar - org.glassfish.hk2:hk2-utils:jar - org.glassfish.hk2:osgi-resource-locator:jar - org.glassfish.hk2:spring-bridge:jar - org.glassfish.jersey.bundles:jaxrs-ri:jar - org.glassfish.jersey.bundles.repackaged:jersey-guava:jar - org.glassfish.jersey.containers:jersey-container-servlet:jar - org.glassfish.jersey.containers:jersey-container-servlet-core:jar - org.glassfish.jersey.core:jersey-client:jar - org.glassfish.jersey.core:jersey-common:jar - org.glassfish.jersey.core:jersey-server:jar - org.glassfish.jersey.ext:jersey-entity-filtering:jar - org.glassfish.jersey.ext:jersey-spring3:jar - org.glassfish.jersey.media:jersey-media-jaxb:jar - org.glassfish.jersey.media:jersey-media-json-jackson:jar - org.glassfish.jersey.media:jersey-media-multipart:jar - org.hdrhistogram:HdrHistogram:jar - org.javassist:javassist:jar - org.json4s:json4s-ast_2.11:jar - org.json4s:json4s-core_2.11:jar - org.json4s:json4s-jackson_2.11:jar - org.jvnet.mimepull:mimepull:jar - org.jvnet:tiger-types:jar - org.latencyutils:LatencyUtils:jar - org.mortbay.jasper:apache-el:jar - org.ow2.asm:asm-analysis:jar - org.ow2.asm:asm-commons:jar - org.ow2.asm:asm-tree:jar - org.reflections:reflections:jar - org.scala-lang.modules:scala-parser-combinators_2.11:jar - org.scala-lang.modules:scala-xml_2.11:jar - org.scala-lang:scala-compiler:jar - org.scala-lang:scala-library:jar - org.scala-lang:scala-reflect:jar - org.scala-lang:scalap:jar - org.slf4j:jul-to-slf4j:jar - org.slf4j:slf4j-api:jar - org.springframework.boot:spring-boot:jar - org.springframework.boot:spring-boot-actuator:jar - org.springframework.boot:spring-boot-actuator-autoconfigure:jar - org.springframework.boot:spring-boot-autoconfigure:jar - org.springframework.boot:spring-boot-starter:jar - org.springframework.boot:spring-boot-starter-actuator:jar - org.springframework.boot:spring-boot-starter-aop:jar - org.springframework.boot:spring-boot-starter-jetty:jar - org.springframework.boot:spring-boot-starter-json:jar - org.springframework.boot:spring-boot-starter-log4j2:jar - org.springframework.boot:spring-boot-starter-web:jar - org.springframework.cloud:spring-cloud-commons:jar - org.springframework.cloud:spring-cloud-config-client:jar - org.springframework.cloud:spring-cloud-context:jar - - org.springframework.cloud:spring-cloud-netflix-core:jar - org.springframework.cloud:spring-cloud-netflix-eureka-client:jar - - org.springframework.cloud:spring-cloud-starter:jar - org.springframework.cloud:spring-cloud-starter-config:jar - org.springframework.cloud:spring-cloud-starter-eureka:jar - org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar - org.springframework.cloud:spring-cloud-starter-netflix-eureka-client:jar - org.springframework.cloud:spring-cloud-starter-netflix-ribbon:jar - org.springframework.security:spring-security-crypto:jar - org.springframework.security:spring-security-rsa:jar - org.springframework:spring-aop:jar - org.springframework:spring-beans:jar - org.springframework:spring-context:jar - org.springframework:spring-core:jar - org.springframework:spring-expression:jar - org.springframework:spring-jcl:jar - org.springframework:spring-web:jar - org.springframework:spring-webmvc:jar - org.yaml:snakeyaml:jar - - xmlpull:xmlpull:jar - xpp3:xpp3_min:jar - - - - - - - ${basedir}/src/main/resources - - * - - 0777 - conf - unix - - - ${basedir}/bin - - * - - 0777 - bin - unix - - - . - - */** - - logs - - - - - diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutionCode.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutionCode.java deleted file mode 100644 index 417da3972..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutionCode.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.entity; - -/** - * Created by peacewong on 2019/11/7. - */ -public class FlowExecutionCode { - - private String version; - - /*private String flowName;*/ - - private Long flowId; - - private Long projectVersionId; - - public String getVersion() { - return version; - } - - public void setVersion(String version) { - this.version = version; - } - - - - - - - - public Long getFlowId() { - return flowId; - } - - public void setFlowId(Long flowId) { - this.flowId = flowId; - } - - public Long getProjectVersionId() { - return projectVersionId; - } - - public void setProjectVersionId(Long projectVersionId) { - this.projectVersionId = projectVersionId; - } - - /* public String getFlowName() { - return flowName; - } - - public void setFlowName(String flowName) { - this.flowName = flowName; - }*/ -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutionNode.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutionNode.java deleted file mode 100644 index 73f6b21a4..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutionNode.java +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.entity; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerNode; - -/** - * Created by peacewong on 2019/11/5. - */ -public class FlowExecutionNode extends AbstractSchedulerNode { -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutonReadNode.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutonReadNode.java deleted file mode 100644 index 386a0b2db..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutonReadNode.java +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.entity; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; - -/** - * Created by peacewong on 2019/11/6. - */ -public class FlowExecutonReadNode extends FlowExecutionNode implements ReadNode { - - private SchedulerNode schedulerNode; - private String[] nodeIds; - - @Override - public String[] getShareNodeIds() { - return this.nodeIds; - } - - @Override - public void setShareNodeIds(String[] strings) { - this.nodeIds = strings; - } - - @Override - public SchedulerNode getSchedulerNode() { - return schedulerNode; - } - - @Override - public void setSchedulerNode(SchedulerNode schedulerNode) { - this.schedulerNode = schedulerNode; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutonSharedNode.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutonSharedNode.java deleted file mode 100644 index 52dfcdcc8..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/FlowExecutonSharedNode.java +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.entity; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; - -/** - * Created by peacewong on 2019/11/6. - */ -public class FlowExecutonSharedNode extends FlowExecutionNode implements ShareNode { - - private SchedulerNode schedulerNode; - private int shareTimes; - - @Override - public int getShareTimes() { - return this.shareTimes; - } - - @Override - public void setShareTimes(int i) { - this.shareTimes = i; - } - - @Override - public SchedulerNode getSchedulerNode() { - return this.schedulerNode; - } - - @Override - public void setSchedulerNode(SchedulerNode schedulerNode) { - this.schedulerNode = schedulerNode; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisJob.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisJob.java deleted file mode 100644 index 6a82646dd..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisJob.java +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.AbstractAppJointLinkisJob; - - -/** - * Created by peacewong on 2019/11/3. - */ -public class FlowExecutionAppJointLinkisJob extends AbstractAppJointLinkisJob { - - - @Override - public String getSubmitUser() { - return getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_SUBMIT_USER()); - } - - - - @Override - public String getUser() { - return getJobProps().get(FlowExecutionEntranceConfiguration.PROXY_USER()); - } - - @Override - public String getJobName() { - return getJobProps().get(FlowExecutionEntranceConfiguration.JOB_ID()); - } - - -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisReadJob.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisReadJob.java deleted file mode 100644 index f1ca1da84..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisReadJob.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration$; -import com.webank.wedatasphere.dss.linkis.node.execution.job.ReadJob; - -import java.util.Map; - -/** - * Created by peacewong on 2019/11/3. - */ -public class FlowExecutionAppJointLinkisReadJob extends FlowExecutionAppJointLinkisJob implements ReadJob { - - private Map sharedNodesInfo; - - private ReadNode readNode; - - public void setReadNode(ReadNode readNode){ - this.readNode = readNode; - } - - @Override - public Map getSharedNodesInfo() { - return this.sharedNodesInfo; - } - - @Override - public String getSharedKey(String value) { - String projectId = getJobProps().get(FlowExecutionEntranceConfiguration.PROJECT_NAME()); - String flowId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_NAME()); - String flowExecId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID()); - return projectId + "." + flowId + "." + flowExecId + "." + value; - } - - @Override - public void setSharedNodesInfo(Map sharedNodesInfo) { - this.sharedNodesInfo = sharedNodesInfo; - } - - @Override - public String[] getShareNodeIds() { - return this.readNode.getShareNodeIds(); - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisSharedJob.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisSharedJob.java deleted file mode 100644 index 1cb725681..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointLinkisSharedJob.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SharedJob; - -/** - * Created by peacewong on 2019/11/3. - */ -public class FlowExecutionAppJointLinkisSharedJob extends FlowExecutionAppJointLinkisJob implements SharedJob { - - private int sharedNum; - - @Override - public int getSharedNum() { - return this.sharedNum; - } - - @Override - public String getSharedKey() { - - String projectId = getJobProps().get(FlowExecutionEntranceConfiguration.PROJECT_NAME()); - String flowId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_NAME()); - String flowExecId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID()); - String nodeId = getJobProps().get(FlowExecutionEntranceConfiguration.JOB_ID()); - return projectId + "." + flowId + "." + flowExecId + "." + nodeId; - - } - - public void setSharedNum(int sharedNum) { - this.sharedNum = sharedNum; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointSignalSharedJob.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointSignalSharedJob.java deleted file mode 100644 index d0d556aad..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppJointSignalSharedJob.java +++ /dev/null @@ -1,56 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.JobSignalKeyCreator; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SignalSharedJob; - -import java.util.Map; - -/** - * Created by johnnwang on 2019/11/14. - */ -public class FlowExecutionAppJointSignalSharedJob extends FlowExecutionAppJointLinkisJob implements SignalSharedJob { - - private JobSignalKeyCreator signalKeyCreator; - - @Override - public JobSignalKeyCreator getSignalKeyCreator() { - return this.signalKeyCreator; - } - - @Override - public void setSignalKeyCreator(JobSignalKeyCreator signalKeyCreator) { - this.signalKeyCreator = signalKeyCreator; - } - - @Override - public String getMsgSaveKey() { - Map configuration = this.getConfiguration(); - Object map = configuration.get("runtime"); - if (map != null && map instanceof Map) { - Map runtime = (Map) map; - Object msgValue = runtime.get(FlowExecutionEntranceConfiguration.EVENT_CHECK_MSG_KEY().getValue()); - if (null != msgValue) { - return msgValue.toString(); - } - } - return null; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisReadJob.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisReadJob.java deleted file mode 100644 index fa094ab37..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisReadJob.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.ReadJob; - -import java.util.Map; - -/** - * Created by peacewong on 2019/11/3. - */ -public class FlowExecutionCommonLinkisReadJob extends FlowExecutionCommonLinkisJob implements ReadJob { - - private Map sharedNodesInfo; - - - private ReadNode readNode; - - public void setReadNode(ReadNode readNode){ - this.readNode = readNode; - } - - @Override - public Map getSharedNodesInfo() { - return this.sharedNodesInfo; - } - - @Override - public String getSharedKey(String value) { - String projectId = getJobProps().get(FlowExecutionEntranceConfiguration.PROJECT_NAME()); - String flowId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_NAME()); - String flowExecId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID()); - return projectId + "." + flowId + "." + flowExecId + "." + value; - } - - @Override - public void setSharedNodesInfo(Map sharedNodesInfo) { - this.sharedNodesInfo = sharedNodesInfo; - } - - @Override - public String[] getShareNodeIds() { - return this.readNode.getShareNodeIds(); - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisSharedJob.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisSharedJob.java deleted file mode 100644 index 81a07d17a..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisSharedJob.java +++ /dev/null @@ -1,47 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SharedJob; - -/** - * Created by peacewong on 2019/11/3. - */ -public class FlowExecutionCommonLinkisSharedJob extends FlowExecutionCommonLinkisJob implements SharedJob { - - private int sharedNum; - - @Override - public int getSharedNum() { - return this.sharedNum; - } - - @Override - public String getSharedKey() { - String projectId = getJobProps().get(FlowExecutionEntranceConfiguration.PROJECT_NAME()); - String flowId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_NAME()); - String flowExecId = getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID()); - String nodeId = getJobProps().get(FlowExecutionEntranceConfiguration.JOB_ID()); - return projectId + "." + flowId + "." + flowExecId + "." + nodeId; - } - - public void setSharedNum(int sharedNum) { - this.sharedNum = sharedNum; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionJobSignalKeyCreator.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionJobSignalKeyCreator.java deleted file mode 100644 index e284b44d8..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionJobSignalKeyCreator.java +++ /dev/null @@ -1,39 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job; - -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.dss.linkis.node.execution.job.JobSignalKeyCreator; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SignalSharedJob; - -public class FlowExecutionJobSignalKeyCreator implements JobSignalKeyCreator { - - @Override - public String getSignalKeyByJob(Job job) { - String projectId = job.getJobProps().get(FlowExecutionEntranceConfiguration.PROJECT_NAME()); - String flowId = job.getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_NAME()); - String flowExecId = job.getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID()); - return projectId + "." + flowId + "." + flowExecId; - } - - @Override - public String getSignalKeyBySignalSharedJob(SignalSharedJob job) { - return getSignalKeyByJob((Job)job); - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionFlowParser.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionFlowParser.java deleted file mode 100644 index fbd8bdd76..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionFlowParser.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractFlowParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.NodeParser; -import com.webank.wedatasphere.dss.common.entity.flow.DSSJSONFlow; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -/** - * Created by peacewong on 2019/11/5. - */ -@Component -public class FlowExecutionFlowParser extends AbstractFlowParser { - - @Autowired - @Override - public void setNodeParsers(NodeParser[] nodeParsers) { - super.setNodeParsers(nodeParsers); - } - - @Override - public Boolean ifFlowCanParse(DSSJSONFlow flow) { - return true; - } - - -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionNodeParser.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionNodeParser.java deleted file mode 100644 index 97c5824b9..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionNodeParser.java +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractNodeParser; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.flow.execution.entrance.entity.FlowExecutionNode; -import org.springframework.stereotype.Component; - - -/** - * Created by peacewong on 2019/11/5. - */ -@Component -public class FlowExecutionNodeParser extends AbstractNodeParser { - - @Override - public SchedulerNode parseNode(DSSNode dssNode) { - FlowExecutionNode node = new FlowExecutionNode(); - node.setDssNode(dssNode); - return node; - } - - @Override - public Boolean ifNodeCanParse(DSSNode dssNode) { - return true; - } - - @Override - public int getOrder() { - return 1; - } - -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionProjectParser.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionProjectParser.java deleted file mode 100644 index 0654c20aa..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionProjectParser.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.parser; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractProjectParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.FlowParser; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -/** - * Created by peacewong on 2019/11/7. - */ -@Component -public class FlowExecutionProjectParser extends AbstractProjectParser { - - @Autowired - @Override - public void setFlowParsers(FlowParser[] flowParsers) { - super.setFlowParsers(flowParsers); - } - - @Override - protected AbstractSchedulerProject createSchedulerProject() { - return new AbstractSchedulerProject() {}; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionReadNodeParser.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionReadNodeParser.java deleted file mode 100644 index 1c04fdfd2..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/parser/FlowExecutionReadNodeParser.java +++ /dev/null @@ -1,55 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.AbstractReadNodeParser; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.flow.execution.entrance.entity.FlowExecutionNode; -import com.webank.wedatasphere.dss.flow.execution.entrance.entity.FlowExecutonReadNode; -import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils; - - -import org.springframework.stereotype.Component; - -/** - * Created by peacewong on 2019/11/7. - */ -@Component -public class FlowExecutionReadNodeParser extends AbstractReadNodeParser { - @Override - public int getOrder() { - return 2; - } - - @Override - public Boolean ifNodeCanParse(DSSNode dssNode) { - return FlowExecutionUtils.isReadNode(dssNode.getNodeType()); - } - - @Override - protected ReadNode createReadNode() { - return new FlowExecutonReadNode(); - } - - @Override - protected SchedulerNode createSchedulerNode() { - return new FlowExecutionNode(); - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/tuning/FlowExecutionFlowTuning.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/tuning/FlowExecutionFlowTuning.java deleted file mode 100644 index 0361e0ded..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/tuning/FlowExecutionFlowTuning.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.AbstractShareNodeFlowTuning; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.NodeTuning; -import com.webank.wedatasphere.dss.flow.execution.entrance.entity.FlowExecutonSharedNode; - -import org.springframework.stereotype.Component; - -/** - * Created by peacewong on 2019/11/7. - */ -@Component -public class FlowExecutionFlowTuning extends AbstractShareNodeFlowTuning { - @Override - public int getOrder() { - return 0; - } - - @Override - public SchedulerFlow tuningSchedulerFlow(SchedulerFlow schedulerFlow) { - schedulerFlow = super.tuningSchedulerFlow(schedulerFlow); - return schedulerFlow; - } - - @Override - public Boolean ifFlowCanTuning(SchedulerFlow schedulerFlow) { - return true ; - } - - @Override - protected ShareNode createShareNode() { - return new FlowExecutonSharedNode(); - } - - @Override - public NodeTuning[] getNodeTunings() { - return new NodeTuning[0]; - } -} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/tuning/FlowExecutionProjectTuning.java b/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/tuning/FlowExecutionProjectTuning.java deleted file mode 100644 index 1ac0ee157..000000000 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/tuning/FlowExecutionProjectTuning.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.tuning; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.AbstractProjectTuning; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.FlowTuning; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -/** - * Created by peacewong on 2019/11/7. - */ -@Component -public class FlowExecutionProjectTuning extends AbstractProjectTuning { - - @Autowired - @Override - public void setFlowTunings(FlowTuning[] flowTunings) { - super.setFlowTunings(flowTunings); - } -} diff --git a/dss-flow-execution-entrance/src/main/resources/application.yml b/dss-flow-execution-entrance/src/main/resources/application.yml deleted file mode 100644 index c4bf9270a..000000000 --- a/dss-flow-execution-entrance/src/main/resources/application.yml +++ /dev/null @@ -1,23 +0,0 @@ -server: - port: 9005 -spring: - application: - name: flowExecutionEntrance - - -eureka: - client: - serviceUrl: - defaultZone: http://127.0.0.1:20303/eureka/ - instance: - metadata-map: - test: wedatasphere - -management: - endpoints: - web: - exposure: - include: refresh,info -logging: - config: classpath:log4j2.xml - diff --git a/dss-flow-execution-entrance/src/main/resources/linkis.properties b/dss-flow-execution-entrance/src/main/resources/linkis.properties deleted file mode 100644 index 09e5c9317..000000000 --- a/dss-flow-execution-entrance/src/main/resources/linkis.properties +++ /dev/null @@ -1,42 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - - -wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.linkis.entrance.restful,com.webank.wedatasphere.dss.flow.execution.entrance.restful - -#wds.linkis.server.component.exclude.classes=com.webank.wedatasphere.linkis.DataWorkCloudApplication - -wds.linkis.engine.application.name=flowExecutionEngine -wds.linkis.enginemanager.application.name=flowExecution - -wds.linkis.query.application.name=cloud-publicservice - -wds.linkis.console.config.application.name=cloud-publicservice -wds.linkis.engine.creation.wait.time.max=20m -wds.linkis.server.version=v1 -#hadoop config dir -#hadoop.config.dir=/appcom/config/hadoop-config -wds.linkis.entrance.config.logPath=file:///tmp/linkis/ - -wds.linkis.resultSet.store.path=hdfs:///tmp/linkis - -wds.linkis.server.socket.mode=true - -##linkis gateway conf -wds.linkis.gateway.url=http://127.0.0.1:9001 -wds.linkis.client.flow.adminuser=ws -wds.linkis.client.flow.author.user.token=WS-AUTH diff --git a/dss-flow-execution-entrance/src/main/resources/log4j.properties b/dss-flow-execution-entrance/src/main/resources/log4j.properties deleted file mode 100644 index 6c6439693..000000000 --- a/dss-flow-execution-entrance/src/main/resources/log4j.properties +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO \ No newline at end of file diff --git a/dss-flow-execution-entrance/src/main/resources/log4j2.xml b/dss-flow-execution-entrance/src/main/resources/log4j2.xml deleted file mode 100644 index 3923cd9f3..000000000 --- a/dss-flow-execution-entrance/src/main/resources/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceConfiguration.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceConfiguration.scala deleted file mode 100644 index 245a23687..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceConfiguration.scala +++ /dev/null @@ -1,95 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.conf - -import com.webank.wedatasphere.linkis.common.conf.{ByteType, CommonVars} - -/** - * created by chaogefeng on 2019/11/4 16:35 - * Description: - */ -object FlowExecutionEntranceConfiguration { - - val APPLICATION_SPRING_APPLICATION = CommonVars("wds.linkis.application.spring.name", "cloud-application") - - val PUBLIC_SERVICE_SPRING_APPLICATION = CommonVars("wds.linkis.publicservice.spring.name", "cloud-publicserivce") - - val GATEWAY_SPRING_APPLICATION = CommonVars("wds.linkis.gateway.spring.name", "dataworkcloud-gateway") - - val QUERY_APPLICATION_NAME = CommonVars("wds.linkis.flow.execution.query.name", "cloud-query") - - val SCHEDULER_APPLICATION = CommonVars("wds.linkis.scheduler.spring.name", "dss-server") - - - val EVENT_CHECK_MSG_KEY = CommonVars("wds.dss.flow.datacheck.msg.key", "msg.savekey") - - val FLOW_EXECUTION_POOL_SIZE = CommonVars("wds.linkis.flow.execution.pool.size", 1000) - - val NODE_PRINT_FLAG = CommonVars("wds.linkis.flow.node.log.flag", false) - - val NODE_STATUS_POLLER_THREAD_SIZE = CommonVars("wds.dds.flow.node.status.poller.thread.size", 20) - - val NODE_STATUS_POLLER_SCHEDULER_TIME = CommonVars("wds.dds.flow.node.status.poller.scheduler.time", 5) - val FLOW_EXECUTION_SCHEDULER_POOL_SIZE = CommonVars("wds.linkis.flow.execution.pool.size", 30) - - val NODE_STATUS_INTERVAL = CommonVars("wds.dds.flow.node.status.poller.interval.time", 3000) - - val COMMAND = "command" - - val JOB_ID = "job.id" - - val FLOW_NAME = "dss.flow.flowid" - - - val PROJECT_NAME = "dss.flow.projectname" - - val FLOW_EXEC_ID = "dss.flow.execid" - - - - val PROXY_USER = "user.to.proxy" - - - val FLOW_SUBMIT_USER = "flow.submituser" - - val READ_NODE_TOKEN = "read.nodes" - - val SHARED_NODE_TOKEN = "share.num" - - - - val SKIP_NODES = CommonVars("wds.dss.flow.skip.nodes", "workflow.subflow") - val EMAIL_TYPE = "linkis.appjoint.sendemail" - - val SIGNAL_NODES = CommonVars("wds.dss.flow.signal.nodes", "linkis.appjoint.eventchecker.eventreceiver") - - val PROPS_MAP = "props_map" - - val FLOW_VAR_MAP = "flow_var_map" - - - val NODE_CONFIGURATION_KEY = "configuration" - - val PROJECT_RESOURCES = "project_resources" - - val FLOW_RESOURCES = "flow_resources" - - - - -} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceSpringConfiguration.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceSpringConfiguration.scala deleted file mode 100644 index 666c7a6e9..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceSpringConfiguration.scala +++ /dev/null @@ -1,60 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.conf - -import com.webank.wedatasphere.dss.flow.execution.entrance.engine.{FlowEntranceEngine, FlowExecutionExecutorManagerImpl} -import com.webank.wedatasphere.dss.flow.execution.entrance.entranceparser.FlowExecutionParser -import com.webank.wedatasphere.linkis.entrance.EntranceParser -import com.webank.wedatasphere.linkis.entrance.annotation.{EngineBuilderBeanAnnotation, EngineManagerBeanAnnotation, EngineRequesterBeanAnnotation, EngineSelectorBeanAnnotation, EntranceExecutorManagerBeanAnnotation, EntranceParserBeanAnnotation, GroupFactoryBeanAnnotation} -import com.webank.wedatasphere.linkis.entrance.execute.{EngineBuilder, EngineManager, EngineRequester, EngineSelector, EntranceExecutorManager, EntranceExecutorRuler} -import com.webank.wedatasphere.linkis.scheduler.queue.GroupFactory -import org.slf4j.LoggerFactory -import org.springframework.beans.factory.annotation.Autowired -import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean -import org.springframework.context.annotation.Configuration - -/** - * created by chaogefeng on 2019/11/4 16:37 - * Description: - */ -@Configuration -class FlowExecutionEntranceSpringConfiguration { - private val logger = LoggerFactory.getLogger(classOf[FlowExecutionEntranceSpringConfiguration]) - - @EntranceExecutorManagerBeanAnnotation - def generateExternalEntranceExecutorManager(@GroupFactoryBeanAnnotation.GroupFactoryAutowiredAnnotation groupFactory: GroupFactory, - @EngineBuilderBeanAnnotation.EngineBuilderAutowiredAnnotation engineBuilder: EngineBuilder, - @EngineRequesterBeanAnnotation.EngineRequesterAutowiredAnnotation engineRequester: EngineRequester, - @EngineSelectorBeanAnnotation.EngineSelectorAutowiredAnnotation engineSelector: EngineSelector, - @EngineManagerBeanAnnotation.EngineManagerAutowiredAnnotation engineManager: EngineManager, - @Autowired entranceExecutorRulers: Array[EntranceExecutorRuler], - @Autowired flowEntranceEngine:FlowEntranceEngine): EntranceExecutorManager = { - logger.info("begin to get FlowExecution Entrance EntranceExecutorManager") - new FlowExecutionExecutorManagerImpl(groupFactory, engineBuilder, engineRequester, engineSelector, engineManager, entranceExecutorRulers, flowEntranceEngine) - } - - - - @EntranceParserBeanAnnotation - def generateEntranceParser(): EntranceParser = { - logger.info("begin to get FlowExecution Entrance parser") - new FlowExecutionParser() - } - - -} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowExecutionExecutorManagerImpl.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowExecutionExecutorManagerImpl.scala deleted file mode 100644 index a3cd00fe7..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowExecutionExecutorManagerImpl.scala +++ /dev/null @@ -1,78 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.engine - -import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob -import com.webank.wedatasphere.linkis.common.utils.Logging -import com.webank.wedatasphere.linkis.entrance.execute.{EngineBuilder, EngineManager, EngineRequester, EngineSelector, EntranceEngine, EntranceExecutorRuler} -import com.webank.wedatasphere.linkis.entrance.execute.impl.EntranceExecutorManagerImpl -import com.webank.wedatasphere.linkis.rpc.Sender -import com.webank.wedatasphere.linkis.scheduler.executer.Executor -import com.webank.wedatasphere.linkis.scheduler.listener.ExecutorListener -import com.webank.wedatasphere.linkis.scheduler.queue.{GroupFactory, Job, SchedulerEvent} - -import scala.concurrent.duration.Duration - -/** - * created by chaogefeng on 2019/11/4 17:25 - * Description: - */ -class FlowExecutionExecutorManagerImpl(groupFactory: GroupFactory, - engineBuilder: EngineBuilder, - engineRequester: EngineRequester, - engineSelector: EngineSelector, - engineManager: EngineManager, - entranceExecutorRulers: Array[EntranceExecutorRuler], - flowEntranceEngine: FlowEntranceEngine - ) - extends EntranceExecutorManagerImpl(groupFactory, engineBuilder, engineRequester, engineSelector, engineManager, entranceExecutorRulers) with Logging{ - - this.flowEntranceEngine.setServiceInstance(Sender.getThisServiceInstance.getApplicationName, Sender.getThisServiceInstance.getInstance) - - override def setExecutorListener(executorListener: ExecutorListener): Unit = super.setExecutorListener(executorListener) - - override def initialEntranceEngine(engine: EntranceEngine): Unit = super.initialEntranceEngine(engine) - - protected override def createExecutor(schedulerEvent: SchedulerEvent): EntranceEngine = schedulerEvent match { - case job: FlowEntranceJob => - - this.flowEntranceEngine - case _ => null - } - - override protected def findExecutors(job: Job): Array[EntranceEngine] = super.findExecutors(job) - - override def askExecutor(schedulerEvent: SchedulerEvent): Option[Executor] = schedulerEvent match { - case job: FlowEntranceJob => - Some(createExecutor(schedulerEvent)) - } - - override def askExecutor(schedulerEvent: SchedulerEvent, wait: Duration): Option[Executor] = schedulerEvent match { - case job: FlowEntranceJob => - Some(createExecutor(schedulerEvent)) - } - - - override def getById(id: Long): Option[Executor] = super.getById(id) - - override def getByGroup(groupName: String): Array[Executor] = super.getByGroup(groupName) - - protected override def delete(executor: Executor): Unit = super.delete(executor) - - override def shutdown(): Unit = super.shutdown() -} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/entranceparser/FlowExecutionParser.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/entranceparser/FlowExecutionParser.scala deleted file mode 100644 index 214502954..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/entranceparser/FlowExecutionParser.scala +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.entranceparser - - -import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob -import com.webank.wedatasphere.linkis.common.utils.Logging -import com.webank.wedatasphere.linkis.entrance.parser.CommonEntranceParser -import com.webank.wedatasphere.linkis.protocol.query.RequestPersistTask -import com.webank.wedatasphere.linkis.protocol.task.Task -import com.webank.wedatasphere.linkis.scheduler.queue.Job -import scala.collection.JavaConversions._ - -/** - * created by chaogefeng on 2019/11/4 17:11 - * Description: - */ -class FlowExecutionParser extends CommonEntranceParser with Logging { - logger.info("FlowExecutionParser Registered") - - override def parseToJob(task: Task): Job = { - val job = new FlowEntranceJob - task match { - case requestPersistTask: RequestPersistTask => - job.setTask(task) - job.setUser(requestPersistTask.getUmUser) - job.setCreator(requestPersistTask.getRequestApplicationName) - job.setParams(requestPersistTask.getParams.toMap[String, Any] ) - job.setEntranceListenerBus(getEntranceContext.getOrCreateEventListenerBus) - job.setListenerEventBus(null) - job.setProgress(0f) - } - job - } -} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobFlowParser.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobFlowParser.scala deleted file mode 100644 index a088e146f..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobFlowParser.scala +++ /dev/null @@ -1,84 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.job.parser - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.{AbstractSchedulerProject, SchedulerFlow} -import com.webank.wedatasphere.dss.common.entity.project.DSSProject -import com.webank.wedatasphere.dss.common.protocol.RequestDSSProject -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration -import com.webank.wedatasphere.dss.flow.execution.entrance.entity.FlowExecutionCode -import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob -import com.webank.wedatasphere.dss.flow.execution.entrance.parser.FlowExecutionProjectParser -import com.webank.wedatasphere.dss.flow.execution.entrance.tuning.FlowExecutionProjectTuning -import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import com.webank.wedatasphere.linkis.rpc.Sender -import org.springframework.beans.factory.annotation.Autowired -import org.springframework.core.annotation.Order -import org.springframework.stereotype.Component - -import scala.collection.JavaConversions._ - - -@Order(1) -@Component -class FlowJobFlowParser extends FlowEntranceJobParser with Logging { - - - @Autowired - var flowExecutionProjectParser: FlowExecutionProjectParser = _ - - @Autowired - var flowExecutionProjectTuning: FlowExecutionProjectTuning = _ - - - override def parse(flowEntranceJob: FlowEntranceJob): Unit = { - info(s"${flowEntranceJob.getId} start to parse flow") - val code = flowEntranceJob.jobToExecuteRequest().code - val flowExecutionCode = LinkisJobExecutionUtils.gson.fromJson(code, classOf[FlowExecutionCode]) - - getDSSProjectByCode(flowExecutionCode) match { - case dwsProject: DSSProject => - - val project = this.flowExecutionProjectParser.parseProject(dwsProject) - - this.flowExecutionProjectTuning.tuningSchedulerProject(project) - - val allFlows = project.asInstanceOf[AbstractSchedulerProject].getProjectVersions.head.getFlows - - - var dwsFlow: SchedulerFlow = null - for (flow <- allFlows) { - if (flowExecutionCode.getFlowId == flow.getId) { - dwsFlow = flow - } - } - //save dwsProject - flowEntranceJob.setDwsProject(dwsProject) - //set flow - flowEntranceJob.setFlow(dwsFlow) - } - info(s"${flowEntranceJob.getId} finished to parse flow") - } - - private def getDSSProjectByCode(flowExecutionCode: FlowExecutionCode) = { - val req = new RequestDSSProject(flowExecutionCode.getFlowId, flowExecutionCode.getVersion, flowExecutionCode.getProjectVersionId) - Sender.getSender(FlowExecutionEntranceConfiguration.SCHEDULER_APPLICATION.getValue).ask(req) - } - -} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/AppJointJobBuilder.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/AppJointJobBuilder.scala deleted file mode 100644 index f8173adfb..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/AppJointJobBuilder.scala +++ /dev/null @@ -1,180 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.node - -import java.{lang, util} - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.{ReadNode, SchedulerNode, ShareNode} -import com.webank.wedatasphere.dss.common.entity.Resource -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration -import com.webank.wedatasphere.dss.flow.execution.entrance.job._ -import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration -import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource -import com.webank.wedatasphere.dss.linkis.node.execution.execution.impl.LinkisNodeExecutionImpl -import com.webank.wedatasphere.dss.linkis.node.execution.job._ -import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParamsParser -import org.apache.commons.lang.StringUtils - -/** - * Created by johnnwang on 2019/11/5. - */ -object AppJointJobBuilder { - - val signalKeyCreator = new FlowExecutionJobSignalKeyCreator - - init() - - def init(): Unit ={ - val jobParamsParser = new JobParamsParser - - jobParamsParser.setSignalKeyCreator(signalKeyCreator) - - LinkisNodeExecutionImpl.getLinkisNodeExecution.asInstanceOf[LinkisNodeExecutionImpl].registerJobParser(jobParamsParser) - } - - def builder():FlowBuilder = new FlowBuilder - - class FlowBuilder extends Builder { - - - private var jobProps: java.util.Map[String, String] = _ - - private var node: SchedulerNode = _ - - def setJobProps(jobProps: java.util.Map[String, String]): FlowBuilder = { - this.jobProps = jobProps - this - } - - def setNode(node: SchedulerNode): FlowBuilder = { - this.node = node - this - } - - override protected def getJobType: String = jobProps.getOrDefault(LinkisJobExecutionConfiguration.LINKIS_TYPE, LinkisJobExecutionConfiguration.LINKIS_DEFAULT_TYPE.getValue(jobProps)) - - override protected def isReadNode: lang.Boolean = this.node.isInstanceOf[ReadNode] - - override protected def isShareNod: lang.Boolean = this.node.isInstanceOf[ShareNode] - - override protected def isSignalSharedNode: lang.Boolean = { - FlowExecutionUtils.isSignalNode(getJobType) - } - - override protected def createReadJob(isLinkisType: Boolean): ReadJob = { - if(isLinkisType){ - val readJob = new FlowExecutionCommonLinkisReadJob - readJob.setReadNode(this.node.asInstanceOf[ReadNode]) - readJob.setJobProps(this.jobProps) - readJob - } else { - val readJob = new FlowExecutionAppJointLinkisReadJob - readJob.setReadNode(this.node.asInstanceOf[ReadNode]) - readJob.setJobProps(this.jobProps) - readJob - } - } - - - override protected def createSharedJob(isLinkisType: Boolean): SharedJob = { - if(isLinkisType){ - val sharedJob = new FlowExecutionCommonLinkisSharedJob - sharedJob.setJobProps(this.jobProps) - sharedJob.setSharedNum(this.node.asInstanceOf[ShareNode].getShareTimes) - sharedJob - } else { - val sharedJob = new FlowExecutionAppJointLinkisSharedJob - sharedJob.setJobProps(this.jobProps) - sharedJob.setSharedNum(this.node.asInstanceOf[ShareNode].getShareTimes) - sharedJob - } - } - - - - override protected def createSignalSharedJob(isLinkisType: Boolean): SignalSharedJob = { - if(isLinkisType){ - null - } else { - val signalJob = new FlowExecutionAppJointSignalSharedJob - signalJob.setSignalKeyCreator(signalKeyCreator) - signalJob.setJobProps(this.jobProps) - signalJob - } - } - - override protected def creatLinkisJob(isLinkisType: Boolean): LinkisJob = { - if(isLinkisType){ - val linkisJob = new FlowExecutionCommonLinkisJob - linkisJob.setJobProps(this.jobProps) - linkisJob - } else { - val linkisJob = new FlowExecutionAppJointLinkisJob - linkisJob.setJobProps(this.jobProps) - linkisJob - } - } - - override protected def fillJobInfo(job: Job): Unit = { - if(StringUtils.isNotEmpty(jobProps.get(FlowExecutionEntranceConfiguration.COMMAND))) - job.setCode(jobProps.get(FlowExecutionEntranceConfiguration.COMMAND)) - job.setParams(new util.HashMap[String,AnyRef]()) - job.setRuntimeParams(new util.HashMap[String,AnyRef]()) - } - - override protected def fillLinkisJobInfo(linkisJob: LinkisJob): Unit = { - this.node.getDssNode.getParams.get(FlowExecutionEntranceConfiguration.NODE_CONFIGURATION_KEY) match { - case configuration:util.Map[String, AnyRef] => - linkisJob.setConfiguration(configuration) - case _ => - } - this.node.getDssNode.getParams.remove(FlowExecutionEntranceConfiguration.FLOW_VAR_MAP) match { - case flowVar:util.Map[String, AnyRef] => - linkisJob.setVariables(flowVar) - case _ => - } - linkisJob.setSource(getSource) - } - - override protected def fillCommonLinkisJobInfo(linkisAppjointJob: CommonLinkisJob): Unit = { - linkisAppjointJob.setJobResourceList(FlowExecutionUtils.resourcesAdaptation(this.node.getDssNode.getResources)) - this.node.getDssNode.getParams.remove(FlowExecutionEntranceConfiguration.PROJECT_RESOURCES) match { - case projectResources:util.List[Resource] => - linkisAppjointJob.setProjectResourceList(FlowExecutionUtils.resourcesAdaptation(projectResources)) - case _ => - } - this.node.getDssNode.getParams.remove(FlowExecutionEntranceConfiguration.FLOW_RESOURCES) match { - case flowResources:util.HashMap[String, util.List[BMLResource]] => - linkisAppjointJob.setFlowNameAndResources(flowResources) - case _ => - } - } - - - private def getSource: util.Map[String, String] = { - val source = new util.HashMap[String, String]() - source.put("projectName", jobProps.get(FlowExecutionEntranceConfiguration.PROJECT_NAME)) - source.put("flowName", jobProps.get(FlowExecutionEntranceConfiguration.FLOW_NAME)) - source.put("nodeName", jobProps.get(FlowExecutionEntranceConfiguration.JOB_ID)) - source - } - - - } -} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/utils/FlowExecutionUtils.scala b/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/utils/FlowExecutionUtils.scala deleted file mode 100644 index 46ad83341..000000000 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/utils/FlowExecutionUtils.scala +++ /dev/null @@ -1,92 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.flow.execution.entrance.utils - -import java.util - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode -import com.webank.wedatasphere.dss.common.entity.Resource -import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration -import com.webank.wedatasphere.dss.linkis.node.execution.WorkflowContext -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration -import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource - -import scala.collection.JavaConversions._ - -/** - * Created by peacewong on 2019/11/6. - */ -object FlowExecutionUtils { - - def isSkippedNode(node:SchedulerNode):Boolean = { - FlowExecutionEntranceConfiguration.SKIP_NODES.getValue.split(",").exists(_.equalsIgnoreCase(node.getNodeType)) - } - - - def isSignalNode(jobType: String) : Boolean = { - FlowExecutionEntranceConfiguration.SIGNAL_NODES.getValue.split(",").exists(_.equalsIgnoreCase(jobType)) - } - - def isAppJointJob(engineType: String): Boolean = LinkisJobExecutionConfiguration.APPJOINT.equalsIgnoreCase(engineType) - - - def getSharedInfo(jobProps: util.Map[String, String], nodeId: String): AnyRef = WorkflowContext.getAppJointContext.getValue(getSharedKey(jobProps, nodeId)) - - def getSharedNodesAndJobId(jobProps: util.Map[String, String], nodes: Array[String]): util.Map[String, AnyRef] = { - val map: util.Map[String, AnyRef] = new util.HashMap[String, AnyRef] - for (nodeId <- nodes) { - map.put(nodeId, getSharedInfo(jobProps, nodeId)) - } - map - } - - def getSharedKey(jobProps: util.Map[String, String], nodeId: String): String = { - val projectId: String = jobProps.get(FlowExecutionEntranceConfiguration.PROJECT_NAME) - val flowId: String = jobProps.get(FlowExecutionEntranceConfiguration.FLOW_NAME) - val flowExecId: String = jobProps.get(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID) - projectId + "." + flowId + "." + flowExecId + "." + nodeId - } - - - def setSharedInfo(jobProps: util.Map[String, String], jobId: String): Unit = { - val nodeId: String = jobProps.get(FlowExecutionEntranceConfiguration.JOB_ID) - val shareNum: Int = jobProps.get(FlowExecutionEntranceConfiguration.SHARED_NODE_TOKEN).toInt - WorkflowContext.getAppJointContext.setValue(getSharedKey(jobProps, nodeId), jobId, shareNum) - } - - def resourcesAdaptation(resources: util.List[Resource]): util.ArrayList[BMLResource] = { - val bmlResources = new util.ArrayList[BMLResource]() - for (resource <- resources){ - bmlResources.add(resourceConvertToBMLResource(resource)) - } - bmlResources - } - - def resourceConvertToBMLResource(resource: Resource):BMLResource = { - val bmlResource = new BMLResource() - bmlResource.setFileName(resource.getFileName) - bmlResource.setResourceId(resource.getResourceId) - bmlResource.setVersion(resource.getVersion) - bmlResource - } - - def isReadNode(nodeType: String): Boolean = { - FlowExecutionEntranceConfiguration.EMAIL_TYPE.equalsIgnoreCase(nodeType) - } - -} diff --git a/dss-framework/dss-appconn-framework/pom.xml b/dss-framework/dss-appconn-framework/pom.xml new file mode 100644 index 000000000..2ab381782 --- /dev/null +++ b/dss-framework/dss-appconn-framework/pom.xml @@ -0,0 +1,127 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + + 4.0.0 + + dss-appconn-framework + + + + junit + junit + 4.12 + test + + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + + com.webank.wedatasphere.dss + dss-appconn-manager-core + ${dss.version} + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + linkis-common + com.webank.wedatasphere.linkis + + + + + com.webank.wedatasphere.linkis + linkis-bml-client + ${linkis.version} + + + commons-beanutils + commons-beanutils + + + linkis-common + com.webank.wedatasphere.linkis + + + json4s-jackson_2.11 + org.json4s + + + + + jakarta.annotation + jakarta.annotation-api + 1.3.5 + provided + + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + + + src/main/java + + **/*.xml + + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/SpringAppConnManager.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/SpringAppConnManager.java new file mode 100644 index 000000000..0a68dbc5d --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/SpringAppConnManager.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn; + +import com.webank.wedatasphere.dss.appconn.manager.impl.AbstractAppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnInfoService; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnResourceService; +import com.webank.wedatasphere.linkis.DataWorkCloudApplication; + + +public class SpringAppConnManager extends AbstractAppConnManager { + + @Override + protected AppConnInfoService createAppConnInfoService() { + return DataWorkCloudApplication.getApplicationContext().getBean(AppConnInfoService.class); + } + + @Override + protected AppConnResourceService createAppConnResourceService() { + return DataWorkCloudApplication.getApplicationContext().getBean(AppConnResourceService.class); + } +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/AppConnMapper.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/AppConnMapper.java new file mode 100644 index 000000000..7eaa59040 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/AppConnMapper.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.dao; + + +import com.webank.wedatasphere.dss.framework.appconn.entity.AppConnBean; +import org.apache.ibatis.annotations.Mapper; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +@Mapper +public interface AppConnMapper { + + /** + * get all appconnbeans + * */ + List getAllAppConnBeans(); + + /** + * get all appconns' name + * */ + List getAllAppConnsName(); + + /** + * get appconnbeans by name + * */ + AppConnBean getAppConnBeanByName(@Param("appConnName") String appConnName); + + /** + * get appconn by id + * */ + AppConnBean getAppConnBeanById(@Param("appConnId") Long appConnId); + + void updateResourceByName(AppConnBean appConnBean); + +} \ No newline at end of file diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/AppInstanceMapper.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/AppInstanceMapper.java new file mode 100644 index 000000000..701506701 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/AppInstanceMapper.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.dao; + + +import com.webank.wedatasphere.dss.framework.appconn.entity.AppInstanceBean; +import org.apache.ibatis.annotations.Mapper; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +@Mapper +public interface AppInstanceMapper { + + /** + * get instance by appconnid + * */ + List getAppInstancesByAppConnId(@Param("appConnId") Long appConnId); + + /** + * get instance by appconnid and label + * */ + List getAppInstance(@Param("appConnId") Long appConnId, + @Param("label") String label); +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/impl/appConnMapper.xml b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/impl/appConnMapper.xml new file mode 100644 index 000000000..1e0a30897 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/impl/appConnMapper.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, + `reference`, `class_name`, `appconn_class_path`,`resource` + + + + `id`, `appconn_name`, `is_user_need_init`, `level`, `if_iframe`, `is_external`, + `reference`, `class_name`, `appconn_class_path`, `resource` + + + + + + + + + + + + update `dss_appconn` + + resource=#{resource}, + appconn_class_path=#{appConnClassPath}, + + where appconn_name=#{appConnName} + + \ No newline at end of file diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/impl/appInstanceMapper.xml b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/impl/appInstanceMapper.xml new file mode 100644 index 000000000..f1cd23c2f --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/dao/impl/appInstanceMapper.xml @@ -0,0 +1,49 @@ + + + + + + + + + + + + + + + + + + + `id`, `appconn_id`, `label`, `url`, `enhance_json`, `homepage_url`, `redirect_url` + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppConnBean.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppConnBean.java new file mode 100644 index 000000000..9e63c8a2b --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppConnBean.java @@ -0,0 +1,155 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.entity; + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.common.entity.Resource; +import java.io.Serializable; +import java.util.List; + + + +public class AppConnBean implements AppConnInfo, Serializable { + + private static final long serialVersionUID=1L; + private Long id; + private String appConnName; + private String isUserNeedInit; + private String level; + private Boolean ifIframe; + private Boolean isExternal; + + // todo:目前通过这两个字段在classpath下加载类。 + // todo:未来这两个字段的作用是在bml和默认应用的。 + private String reference; + private String resource; + private Resource resourceObj; + private String className; + private String appConnClassPath; + + private List appInstanceBeans; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + @Override + public String getAppConnName() { + return appConnName; + } + + public void setAppConnName(String appConnName) { + this.appConnName = appConnName; + } + + public String getIsUserNeedInit() { + return isUserNeedInit; + } + + public void setIsUserNeedInit(String isUserNeedInit) { + this.isUserNeedInit = isUserNeedInit; + } + + public String getLevel() { + return level; + } + + public void setLevel(String level) { + this.level = level; + } + + public Boolean getIfIframe() { + return ifIframe; + } + + public void setIfIframe(Boolean ifIframe) { + this.ifIframe = ifIframe; + } + + public Boolean getExternal() { + return isExternal; + } + + public void setExternal(Boolean external) { + isExternal = external; + } + + public String getReference() { + return reference; + } + + public void setReference(String reference) { + this.reference = reference; + } + + public String getResource() { + return resource; + } + + public void setResource(String resource) { + this.resource = resource; + } + + public List getAppInstanceBeans() { + return appInstanceBeans; + } + + public void setAppInstanceBeans(List appInstanceBeans) { + this.appInstanceBeans = appInstanceBeans; + } + + @Override + public String getClassName() { + return className; + } + + @Override + public Resource getAppConnResource() { + return resourceObj; + } + + public void setAppConnResource(Resource resource) { + resourceObj = resource; + } + + public void setClassName(String className) { + this.className = className; + } + + public String getAppConnClassPath() { + return appConnClassPath; + } + + public void setAppConnClassPath(String appConnClassPath) { + this.appConnClassPath = appConnClassPath; + } + + @Override + public String toString() { + return "AppConnBean{" + + "id=" + id + + ", appConnName='" + appConnName + '\'' + + ", className='" + className + '\'' + + ", appConnClassPath='" + appConnClassPath + '\'' + + '}'; + } +} \ No newline at end of file diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppConnResource.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppConnResource.java new file mode 100644 index 000000000..a698e6bc8 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppConnResource.java @@ -0,0 +1,51 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.entity; + +import com.webank.wedatasphere.dss.common.entity.Resource; + + +public class AppConnResource { + + private Resource resource; + private long lastModifiedTime; + private long size; + + public Resource getResource() { + return resource; + } + + public void setResource(Resource resource) { + this.resource = resource; + } + + public long getLastModifiedTime() { + return lastModifiedTime; + } + + public void setLastModifiedTime(long lastModifiedTime) { + this.lastModifiedTime = lastModifiedTime; + } + + public long getSize() { + return size; + } + + public void setSize(long size) { + this.size = size; + } +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppInstanceBean.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppInstanceBean.java new file mode 100644 index 000000000..1914a0a29 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/entity/AppInstanceBean.java @@ -0,0 +1,110 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.entity; + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppInstanceInfo; +import java.io.Serializable; + + +public class AppInstanceBean implements AppInstanceInfo, Serializable { + + private static final long serialVersionUID=1L; + + private Long id; + private Long appConnId; + private String label; + private String url; + private String enhanceJson; + private String homepageUrl; + private String redirectUrl; + + @Override + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getAppconnId() { + return appConnId; + } + + public void setAppConnId(Long appConnId) { + this.appConnId = appConnId; + } + + public String getLabel() { + return label; + } + + public void setLabel(String label) { + this.label = label; + } + + @Override + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + @Override + public String getEnhanceJson() { + return enhanceJson; + } + + public void setEnhanceJson(String enhanceJson) { + this.enhanceJson = enhanceJson; + } + + @Override + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + @Override + public String getRedirectUrl() { + return redirectUrl; + } + + @Override + public String getLabels() { + return label; + } + + public void setRedirectUrl(String redirectUrl) { + this.redirectUrl = redirectUrl; + } + + @Override + public String toString() { + return "AppInstanceBean{" + + "id=" + id + + ", appConnId=" + appConnId + + ", label='" + label + '\'' + + ", url='" + url + '\'' + + '}'; + } +} \ No newline at end of file diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/exception/AppConnNotExistsErrorException.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/exception/AppConnNotExistsErrorException.java new file mode 100644 index 000000000..d73dfcb3c --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/exception/AppConnNotExistsErrorException.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.exception; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + + +public class AppConnNotExistsErrorException extends DSSErrorException { + + public AppConnNotExistsErrorException(int errCode, String desc) { + super(errCode, desc); + } + + public AppConnNotExistsErrorException(int errCode, String desc, Throwable cause) { + super(errCode, desc); + initCause(cause); + } +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/restful/AppConnManagerRestfulApi.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/restful/AppConnManagerRestfulApi.java new file mode 100644 index 000000000..e9aabdb31 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/restful/AppConnManagerRestfulApi.java @@ -0,0 +1,106 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.restful; + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppInstanceInfo; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnInfoService; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.framework.appconn.service.AppConnResourceUploadService; +import com.webank.wedatasphere.linkis.server.Message; +import java.util.List; +import javax.annotation.PostConstruct; +import javax.ws.rs.Consumes; +import javax.ws.rs.GET; +import javax.ws.rs.Path; +import javax.ws.rs.PathParam; +import javax.ws.rs.Produces; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import org.apache.commons.lang.exception.ExceptionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +@Component +@Path("/dss/framework/project/appconn") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class AppConnManagerRestfulApi { + + private static final Logger LOGGER = LoggerFactory.getLogger(AppConnManagerRestfulApi.class); + + @Autowired + private AppConnInfoService appConnInfoService; + @Autowired + private AppConnResourceUploadService appConnResourceUploadService; + + @PostConstruct + public void init() { + LOGGER.info("Try to scan AppConn plugins..."); + appConnInfoService.getAppConnInfos().forEach(DSSExceptionUtils.handling(appConnInfo -> { + LOGGER.info("Try to load or update AppConn {}.", appConnInfo.getAppConnName()); + appConnResourceUploadService.upload(appConnInfo.getAppConnName()); + })); + LOGGER.info("All AppConn plugins has scanned."); + } + + @GET + @Path("listAppConnInfos") + public Response listAppConnInfos() { + List appConnInfos = appConnInfoService.getAppConnInfos(); + Message message = Message.ok("Get AppConnInfo list succeed."); + message.data("appConnInfos", appConnInfos); + return Message.messageToResponse(message); + } + + @GET + @Path("{appConnName}/get") + public Response get(@PathParam("appConnName") String appConnName) { + AppConnInfo appConnInfo = appConnInfoService.getAppConnInfo(appConnName); + Message message = Message.ok("Get AppConnInfo succeed."); + message.data("appConnInfo", appConnInfo); + return Message.messageToResponse(message); + } + + @GET + @Path("/{appConnName}/getAppInstances") + public Response getAppInstancesByAppConnInfo(@PathParam("appConnName") String appConnName) { + List appInstanceInfos = appConnInfoService.getAppInstancesByAppConnName(appConnName); + Message message = Message.ok("Get AppInstance list succeed."); + message.data("appInstanceInfos", appInstanceInfos); + return Message.messageToResponse(message); + } + + @GET + @Path("/{appConnName}/load") + public Response load(@PathParam("appConnName") String appConnName) { + LOGGER.info("Try to load a new AppConn {}.", appConnName); + try { + appConnResourceUploadService.upload(appConnName); + } catch (Exception e) { + LOGGER.error("Load AppConn " + appConnName + " failed.", e); + Message message = Message.error("Load AppConn " + appConnName + " failed. Reason: " + ExceptionUtils.getRootCauseMessage(e)); + return Message.messageToResponse(message); + } + Message message = Message.ok("Load AppConn " + appConnName + " succeed."); + return Message.messageToResponse(message); + } + +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/AppConnResourceUploadService.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/AppConnResourceUploadService.java new file mode 100644 index 000000000..f97951b77 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/AppConnResourceUploadService.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.service; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + + +public interface AppConnResourceUploadService { + + void upload(String appConnName) throws DSSErrorException; + +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/impl/AppConnInfoServiceImpl.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/impl/AppConnInfoServiceImpl.java new file mode 100644 index 000000000..c15079178 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/impl/AppConnInfoServiceImpl.java @@ -0,0 +1,69 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.service.impl; + +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppInstanceInfo; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnInfoService; +import com.webank.wedatasphere.dss.framework.appconn.dao.AppConnMapper; +import com.webank.wedatasphere.dss.framework.appconn.dao.AppInstanceMapper; +import com.webank.wedatasphere.dss.framework.appconn.entity.AppConnBean; +import com.webank.wedatasphere.dss.framework.appconn.utils.AppConnServiceUtils; +import java.util.List; +import org.apache.commons.lang.StringUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +@Component +public class AppConnInfoServiceImpl implements AppConnInfoService { + + @Autowired + private AppConnMapper appConnMapper; + + @Autowired + private AppInstanceMapper appInstanceMapper; + + @Override + public List getAppConnInfos() { + List appConnBeans = appConnMapper.getAllAppConnBeans(); + appConnBeans.forEach(appConnBean -> { + String resource = appConnBean.getResource(); + if(StringUtils.isNotBlank(resource)) { + appConnBean.setAppConnResource(AppConnServiceUtils.stringToResource(resource).getResource()); + } + }); + return appConnBeans; + } + + @Override + public AppConnInfo getAppConnInfo(String appConnName) { + return appConnMapper.getAppConnBeanByName(appConnName); + } + + @Override + public List getAppInstancesByAppConnInfo(AppConnInfo appConnInfo) { + Long id = ((AppConnBean) appConnInfo).getId(); + return appInstanceMapper.getAppInstancesByAppConnId(id); + } + + @Override + public List getAppInstancesByAppConnName(String appConnName) { + AppConnBean appConnBean = appConnMapper.getAppConnBeanByName(appConnName); + return getAppInstancesByAppConnInfo(appConnBean); + } +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/impl/AppConnResourceServiceImpl.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/impl/AppConnResourceServiceImpl.java new file mode 100644 index 000000000..2d8ff0d5b --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/service/impl/AppConnResourceServiceImpl.java @@ -0,0 +1,173 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.service.impl; + +import com.webank.wedatasphere.dss.appconn.loader.utils.AppConnUtils; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.entity.AppConnInfo; +import com.webank.wedatasphere.dss.appconn.manager.impl.AbstractAppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.service.AppConnResourceService; +import com.webank.wedatasphere.dss.appconn.manager.utils.AppConnIndexFileUtils; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.ZipHelper; +import com.webank.wedatasphere.dss.framework.appconn.dao.AppConnMapper; +import com.webank.wedatasphere.dss.framework.appconn.entity.AppConnBean; +import com.webank.wedatasphere.dss.framework.appconn.entity.AppConnResource; +import com.webank.wedatasphere.dss.framework.appconn.exception.AppConnNotExistsErrorException; +import com.webank.wedatasphere.dss.framework.appconn.service.AppConnResourceUploadService; +import com.webank.wedatasphere.dss.framework.appconn.utils.AppConnServiceUtils; +import com.webank.wedatasphere.linkis.bml.client.BmlClient; +import com.webank.wedatasphere.linkis.bml.client.BmlClientFactory; +import com.webank.wedatasphere.linkis.bml.protocol.BmlUpdateResponse; +import com.webank.wedatasphere.linkis.bml.protocol.BmlUploadResponse; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import javax.annotation.PostConstruct; +import javax.annotation.PreDestroy; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +@Component +public class AppConnResourceServiceImpl implements AppConnResourceService, AppConnResourceUploadService { + + private Logger LOGGER = LoggerFactory.getLogger(AppConnResourceServiceImpl.class); + private static final String LIB_NAME = "/lib"; + + @Autowired + private AppConnMapper appConnMapper; + private BmlClient bmlClient; + + @PostConstruct + public void init() { + bmlClient = BmlClientFactory.createBmlClient(); + } + + @Override + public String getAppConnHome(AppConnInfo appConnInfo) { + AppConnBean appConnBean = (AppConnBean) appConnInfo; + String appConnHome=appConnBean.getAppConnClassPath(); + if(appConnHome.endsWith(LIB_NAME)){ + appConnHome =appConnHome.substring(0,appConnHome.lastIndexOf(LIB_NAME)); + } + return appConnHome; + } + + @Override + public void upload(String appConnName) throws DSSErrorException { + File appConnPath = new File(AppConnUtils.getAppConnHomePath(), appConnName); + if (!appConnPath.exists()) { + throw new AppConnNotExistsErrorException(20350, "AppConn home path " + appConnPath.getPath() + " not exists."); + } else if (!appConnPath.isDirectory()) { + throw new AppConnNotExistsErrorException(20350, "AppConn home path " + appConnPath.getPath() + " is not a directory."); + } + File zipFile = new File(appConnPath.getPath() + ".zip"); + if (zipFile.exists() && !zipFile.delete()) { + throw new AppConnNotExistsErrorException(20001, "No permission to delete old zip file " + zipFile); + } + ZipHelper.zip(appConnPath.getPath(), false); +// TODO ZipUtils.fileToZip(appConnPath.getPath(), AppConnUtils.getAppConnHomePath(), appConnName + ".zip"); + AppConnBean appConnBean = appConnMapper.getAppConnBeanByName(appConnName); + AppConnResource appConnResource; + File indexFile = null; + if (appConnBean == null) { + throw new AppConnNotExistsErrorException(20350, "AppConn not exists in DB, please update db at first."); + } else if (StringUtils.isNotBlank(appConnBean.getResource())) { + // If resource is exists, then indexFile is also exists. + appConnResource = AppConnServiceUtils.stringToResource(appConnBean.getResource()); + indexFile = AppConnIndexFileUtils.getIndexFile(appConnPath); + if (appConnPath.lastModified() == appConnResource.getLastModifiedTime() + && zipFile.length() == appConnResource.getSize() && AppConnIndexFileUtils.isLatestIndex(appConnPath, appConnResource.getResource())) { + LOGGER.info("No necessary to update the AppConn {}, since it's packages has no changes in path {}.", appConnName, appConnPath.getPath()); + return; + } + } else { + // If resource is not exists, this is the first time to upload this AppConn. + appConnResource = new AppConnResource(); + } + // At first, upload appConn file to BML + Resource resource = new Resource(); + InputStream inputStream = null; + if (appConnResource.getResource() != null) { + try { + inputStream = new FileInputStream(zipFile.getPath()); + + BmlUpdateResponse response = bmlClient.updateResource(Utils.getJvmUser(), appConnResource.getResource().getResourceId(), zipFile.getPath(),inputStream); + resource.setResourceId(appConnResource.getResource().getResourceId()); + resource.setVersion(response.version()); + } catch (FileNotFoundException e) { + throw new AppConnNotExistsErrorException(20351, "AppConn update to bml failed"+e.getMessage()); + } finally { + IOUtils.closeQuietly(inputStream); + } + } else { + + try { + inputStream = new FileInputStream(zipFile.getPath()); + BmlUploadResponse response = bmlClient.uploadResource(Utils.getJvmUser(), zipFile.getPath(), inputStream); + resource.setResourceId(response.resourceId()); + resource.setVersion(response.version()); + } catch (FileNotFoundException e) { + throw new AppConnNotExistsErrorException(20352, "AppConn update to bml failed"+e.getMessage()); + } finally { + IOUtils.closeQuietly(inputStream); + } + + } + resource.setFileName(zipFile.getName()); + // Then, insert into db. + appConnResource.setLastModifiedTime(appConnPath.lastModified()); + appConnResource.setSize(zipFile.length()); + appConnResource.setResource(resource); + String resourceStr = AppConnServiceUtils.resourceToString(appConnResource); + + AppConnBean appConnBeanReLoad = new AppConnBean(); + appConnBeanReLoad.setId(appConnBean.getId()); + appConnBeanReLoad.setResource(resourceStr); + appConnBeanReLoad.setAppConnName(appConnName); + appConnBeanReLoad.setAppConnClassPath(appConnPath.getPath()); + appConnBeanReLoad.setClassName(appConnBean.getClassName()); + appConnMapper.updateResourceByName(appConnBeanReLoad); + // update index file. + if (indexFile != null && !indexFile.delete()) { + throw new AppConnNotExistsErrorException(20350, "Delete index file " + indexFile.getName() + " failed, please ensure the permission is all right."); + } + indexFile = new File(appConnPath, AppConnIndexFileUtils.getIndexFileName(resource)); + try { + indexFile.createNewFile(); + } catch (IOException e) { + throw new AppConnNotExistsErrorException(20350, "create index file " + indexFile.getName() + " failed, please ensure the permission is all right.", e); + } + // Finally, reload this AppConn. + ((AbstractAppConnManager) AppConnManager.getAppConnManager()).reloadAppConn(appConnBeanReLoad); + LOGGER.info("AppConn {} has updated resource to {}.", appConnName, resourceStr); + } + + @PreDestroy + public void destory() { + IOUtils.closeQuietly(bmlClient); + } +} diff --git a/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/utils/AppConnServiceUtils.java b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/utils/AppConnServiceUtils.java new file mode 100644 index 000000000..11d851cd8 --- /dev/null +++ b/dss-framework/dss-appconn-framework/src/main/java/com/webank/wedatasphere/dss/framework/appconn/utils/AppConnServiceUtils.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.appconn.utils; + +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.framework.appconn.entity.AppConnResource; + + +public class AppConnServiceUtils { + + public static String resourceToString(AppConnResource resource) { + return DSSCommonUtils.COMMON_GSON.toJson(resource); + } + + public static AppConnResource stringToResource(String resource) { + return DSSCommonUtils.COMMON_GSON.fromJson(resource, AppConnResource.class); + } + +} diff --git a/dss-framework/dss-framework-common/pom.xml b/dss-framework/dss-framework-common/pom.xml new file mode 100644 index 000000000..05aac9fe1 --- /dev/null +++ b/dss-framework/dss-framework-common/pom.xml @@ -0,0 +1,115 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + + 4.0.0 + dss-framework-common + + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + org.springframework.cloud + spring-cloud-netflix + + + spring-cloud-starter-netflix-eureka-client + org.springframework.cloud + + + + + org.apache.commons + commons-math3 + provided + + + xstream + com.thoughtworks.xstream + + + org.springframework.cloud + spring-cloud-starter-netflix-eureka-client + ${spring.cloud.version} + provided + + + logback-classic + ch.qos.logback + + + log4j-to-slf4j + org.apache.logging.log4j + + + gson + com.google.code.gson + + + jsr311-api + javax.ws.rs + + + xstream + com.thoughtworks.xstream + + + commons-math + org.apache.commons + + + jackson-core + com.fasterxml.jackson.core + + + spring-boot-autoconfigure + org.springframework.boot + + + spring-boot-starter-aop + org.springframework.boot + + + spring-boot-starter + org.springframework.boot + + + spring-boot-starter-cache + org.springframework.boot + + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkErrorException.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkErrorException.java new file mode 100644 index 000000000..2a0e29133 --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkErrorException.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class DSSFrameworkErrorException extends ErrorException { + + public DSSFrameworkErrorException(int errorCode, String errorDesc){ + super(errorCode, errorDesc); + } + + public static void dealErrorException(int errorCode, String errorDesc, Throwable t) throws DSSFrameworkErrorException{ + DSSFrameworkErrorException dssFrameworkErrorException = new DSSFrameworkErrorException(errorCode, errorDesc); + dssFrameworkErrorException.initCause(t); + throw dssFrameworkErrorException; + } + + public static void dealErrorException(int errorCode, String errorDesc) throws DSSFrameworkErrorException{ + throw new DSSFrameworkErrorException(errorCode, errorDesc); + } + + +} diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkRuntimeException.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkRuntimeException.java new file mode 100644 index 000000000..1e1df42eb --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkRuntimeException.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.exception; + +import com.webank.wedatasphere.linkis.common.exception.WarnException; + + +public class DSSFrameworkRuntimeException extends WarnException { + + public DSSFrameworkRuntimeException(String msg){ + super(100000, msg); + } +} diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkWarnException.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkWarnException.java new file mode 100644 index 000000000..b41db5bc9 --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/DSSFrameworkWarnException.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.exception; + +import com.webank.wedatasphere.linkis.common.exception.WarnException; + + +public class DSSFrameworkWarnException extends WarnException { + public DSSFrameworkWarnException(int errorCode, String errorDesc){ + super(errorCode, errorDesc); + } + + public static void dealWarnException(int errorCode, String errorDesc, Throwable t) throws DSSFrameworkWarnException { + DSSFrameworkWarnException dssFrameworkWarnException = new DSSFrameworkWarnException(errorCode, errorDesc); + dssFrameworkWarnException.initCause(t); + throw dssFrameworkWarnException; + } + +} diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/ThrowingConsumer.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/ThrowingConsumer.java new file mode 100644 index 000000000..56cf17087 --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/ThrowingConsumer.java @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.exception; + + +@FunctionalInterface +public interface ThrowingConsumer { + void accept(T t) throws E; +} diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/ThrowingFunction.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/ThrowingFunction.java new file mode 100644 index 000000000..54e1f3b37 --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/exception/ThrowingFunction.java @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.exception; + + +@FunctionalInterface +public interface ThrowingFunction { + R accept(T t) throws E; +} diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/utils/DSSFrameworkExceptionUtils.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/utils/DSSFrameworkExceptionUtils.java new file mode 100644 index 000000000..24b846893 --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/utils/DSSFrameworkExceptionUtils.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.utils; + + +import com.webank.wedatasphere.dss.framework.common.exception.DSSFrameworkRuntimeException; +import com.webank.wedatasphere.dss.framework.common.exception.ThrowingConsumer; +import com.webank.wedatasphere.dss.framework.common.exception.ThrowingFunction; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.function.Consumer; +import java.util.function.Function; + + +public class DSSFrameworkExceptionUtils { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSFrameworkExceptionUtils.class); + + public static Consumer handling( + ThrowingConsumer throwingConsumer) { + return i -> { + try { + throwingConsumer.accept(i); + } catch (Exception e) { + LOGGER.error("execute failed,reason:",e); + throw new DSSFrameworkRuntimeException(e.getMessage()); + } + }; + } + + public static Function map( + ThrowingFunction throwingFunction) { + return i -> { + try { + return throwingFunction.accept(i); + } catch (Exception e) { + LOGGER.error("execute failed,reason:",e); + throw new DSSFrameworkRuntimeException(e.getMessage()); + } + }; + } +} diff --git a/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/utils/RestfulUtils.java b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/utils/RestfulUtils.java new file mode 100644 index 000000000..5e9b7d185 --- /dev/null +++ b/dss-framework/dss-framework-common/src/main/java/com/webank/wedatasphere/dss/framework/common/utils/RestfulUtils.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.common.utils; + +import com.webank.wedatasphere.linkis.server.Message; +import org.apache.commons.math3.util.Pair; + +import javax.ws.rs.core.Response; +import java.util.Arrays; + + +public class RestfulUtils { + + + public static Response dealError(String reason){ + Message message = Message.error(reason); + return Message.messageToResponse(message); + } + + public static Response dealOk(String msg){ + Message message = Message.ok(msg); + return Message.messageToResponse(message); + } + + + + @SafeVarargs + public static Response dealOk(String msg, Pair... data){ + Message message = Message.ok(msg); + Arrays.stream(data).forEach(p -> message.data(p.getKey(), p.getValue())); + return Message.messageToResponse(message); + } + +} diff --git a/dss-framework/dss-framework-orchestrator-server/pom.xml b/dss-framework/dss-framework-orchestrator-server/pom.xml new file mode 100644 index 000000000..d847d188d --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/pom.xml @@ -0,0 +1,187 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + + 4.0.0 + + dss-framework-orchestrator-server + + + + com.webank.wedatasphere.dss + dss-framework-common + ${dss.version} + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + + + commons-beanutils + commons-beanutils + + + commons-collections + commons-collections + + + + + com.webank.wedatasphere.dss + dss-orchestrator-db + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-orchestrator-loader + ${dss.version} + + + com.webank.wedatasphere.dss + dss-framework-orchestrator-publish + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + spring-cloud-starter-netflix-eureka-client + org.springframework.cloud + + + + + + com.webank.wedatasphere.dss + dss-contextservice + ${dss.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + commons-text + org.apache.commons + + + json4s-jackson_2.11 + org.json4s + + + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-appconn-manager-client + ${dss.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + + 8 + 8 + + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + **/*.properties + **/*.yml + + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/assembly/distribution.xml b/dss-framework/dss-framework-orchestrator-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..313ea1281 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/assembly/distribution.xml @@ -0,0 +1,44 @@ + + + + dss-framework-orchestrator-server + + dir + + true + dss-framework-orchestrator-server + + + + + + lib + true + true + false + true + true + + + + + + + diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/conf/OrchestratorConf.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/conf/OrchestratorConf.java new file mode 100644 index 000000000..4bef87f06 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/conf/OrchestratorConf.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public class OrchestratorConf { + public static final CommonVars DSS_UPLOAD_PATH = CommonVars.apply("wds.dss.file.upload.dir", "/appcom/tmp/uploads"); +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/conf/OrchestratorSpringConf.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/conf/OrchestratorSpringConf.java new file mode 100644 index 000000000..e2df01daa --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/conf/OrchestratorSpringConf.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.conf; + +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.contextservice.service.impl.ContextServiceImpl; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + + +@Configuration +public class OrchestratorSpringConf { + + @Bean(name = "contextService") + public ContextService createContextService(){ + return ContextServiceImpl.getInstance(); + } + +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/constant/DSSOrchestratorConstant.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/constant/DSSOrchestratorConstant.java new file mode 100644 index 000000000..81a5bb06b --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/constant/DSSOrchestratorConstant.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.constant; + + +public class DSSOrchestratorConstant { + public static final String PUBLISH_FLOW_REPORT_FORMATE = "工作流名:%s,版本号:%s，工作流内容为空,请自行修改或者删除"; +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/constant/OrchestratorSpringConfiguration.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/constant/OrchestratorSpringConfiguration.java new file mode 100644 index 000000000..f9176d6c6 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/constant/OrchestratorSpringConfiguration.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.constant; + +import com.webank.wedatasphere.dss.orchestrator.server.service.OrchestratorPluginService; +import com.webank.wedatasphere.dss.orchestrator.server.service.impl.OrchestratorPluginServiceImpl; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class OrchestratorSpringConfiguration { + + @Bean + @ConditionalOnMissingBean + public OrchestratorPluginService createOrchestratorPluginService() { + return new OrchestratorPluginServiceImpl(); + } + +} \ No newline at end of file diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/entity/UploadFileResponse.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/entity/UploadFileResponse.java new file mode 100644 index 000000000..d1d72a6c1 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/entity/UploadFileResponse.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.entity; + +public class UploadFileResponse { + private String fileName; + private String fileDownloadUri; + private String fileType; + private long size; + + public UploadFileResponse(String fileName, String fileDownloadUri, String fileType, long size) { + this.fileName = fileName; + this.fileDownloadUri = fileDownloadUri; + this.fileType = fileType; + this.size = size; + } + // getter and setter ... +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/entity/query/QueryOrchestratorVersion.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/entity/query/QueryOrchestratorVersion.java new file mode 100644 index 000000000..110d021e8 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/entity/query/QueryOrchestratorVersion.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.entity.query; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; + +@XmlRootElement +public class QueryOrchestratorVersion { + + @NotNull(message = "orchestratorId不能为空") + private Long orchestratorId; + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/restful/OrchestratorIERestful.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/restful/OrchestratorIERestful.java new file mode 100644 index 000000000..465c75225 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/restful/OrchestratorIERestful.java @@ -0,0 +1,181 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.restful; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.label.LabelKeyConvertor; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext; +import com.webank.wedatasphere.dss.orchestrator.core.service.BMLService; +import com.webank.wedatasphere.dss.orchestrator.publish.ExportDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.publish.ImportDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.server.service.OrchestratorService; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.io.IOUtils; +import org.glassfish.jersey.media.multipart.FormDataBodyPart; +import org.glassfish.jersey.media.multipart.FormDataContentDisposition; +import org.glassfish.jersey.media.multipart.FormDataMultiPart; +import org.glassfish.jersey.media.multipart.FormDataParam; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.io.IOException; +import java.io.InputStream; +import java.io.UnsupportedEncodingException; +import java.util.Arrays; +import java.util.List; +import java.util.Map; + +@Produces(MediaType.APPLICATION_JSON) +@Consumes({MediaType.APPLICATION_JSON, MediaType.MULTIPART_FORM_DATA}) +@Component +@Path("/dss/framework/orchestrator") +public class OrchestratorIERestful { + private static final Logger logger = LoggerFactory.getLogger(OrchestratorIERestful.class); + @Autowired + private BMLService bmlService; + @Autowired + OrchestratorService orchestratorService; + @Autowired + private DSSOrchestratorContext orchestratorContext; + + @POST + @Path("/importOrchestratorFile") + public Response importOrcFile(@Context HttpServletRequest req, + @FormDataParam("projectName") String projectName, + @FormDataParam("projectID") Long projectID, + @FormDataParam("labels") String labels, + FormDataMultiPart form) throws DSSErrorException, UnsupportedEncodingException { + List files = form.getFields("file"); + if (null == files || files.size() == 0) { + throw new DSSErrorException(100788, "Import orchestrator failed for files is empty"); + } + Long importOrcId = 0L; + for (FormDataBodyPart p : files) { + InputStream inputStream = p.getValueAs(InputStream.class); + FormDataContentDisposition fileDetail = p.getFormDataContentDisposition(); + String fileName = new String(fileDetail.getFileName().getBytes("ISO8859-1"), "UTF-8"); + String userName = SecurityFilter.getLoginUsername(req); + //调用工具类生产label + List dssLabelList = getDSSLabelList(labels); +// String fileDownloadUri = ServletUriComponentsBuilder.fromCurrentContextPath() +// .path("/downloadFile/") +// .path(fileName) +// .toUriString(); +// java.nio.file.Path targetLocation = DSSFileService.fileStorageLocation.resolve(fileName); + Workspace workspace = SSOHelper.getWorkspace(req); + //3、打包新的zip包上传BML +// InputStream inputStream = bmlService.readLocalResourceFile(userName, targetLocation.toAbsolutePath().toString()); + Map resultMap = bmlService.upload(userName, inputStream, + fileName, projectName); + try { + importOrcId = orchestratorContext.getDSSOrchestratorPlugin(ImportDSSOrchestratorPlugin.class).importOrchestrator(userName, + workspace.getWorkspaceName(), + projectName, + projectID, + resultMap.get("resourceId").toString(), + resultMap.get("version").toString(), + dssLabelList, + workspace); + } catch (Exception e) { + logger.error("Import orchestrator failed for ", e); + throw new DSSErrorException(100789, "Import orchestrator failed for " + e.getMessage()); + } + } + return Message.messageToResponse(Message.ok().data("importOrcId", importOrcId)); + } + + @GET + @Path("/exportOrchestrator") + public void exportOrcFile(@Context HttpServletRequest req, + @Context HttpServletResponse resp, + @DefaultValue("exportOrc") @QueryParam("outputFileName") String outputFileName, + @DefaultValue("utf-8") @QueryParam("charset") String charset, + @DefaultValue("zip") @QueryParam("outputFileType") String outputFileType, + @QueryParam("projectName") String projectName, + @QueryParam("orchestratorId") Long orchestratorId, + @QueryParam("orcVersionId") Long orcVersionId, + @DefaultValue("false") @QueryParam("addOrcVersion") Boolean addOrcVersion, + @QueryParam("labels") String labels) throws DSSErrorException, IOException { + resp.addHeader("Content-Disposition", "attachment;filename=" + + new String(outputFileName.getBytes("UTF-8"), "ISO8859-1") + "." + outputFileType); + resp.setCharacterEncoding(charset); + Workspace workspace = SSOHelper.getWorkspace(req); + String userName = SecurityFilter.getLoginUsername(req); + List dssLabelList = getDSSLabelList(labels); + Map res = null; + OrchestratorVo orchestratorVo = orchestratorService.getOrchestratorVoById(orchestratorId); + orcVersionId = orchestratorVo.getDssOrchestratorVersion().getId(); + logger.info("export orchestrator orchestratorId " + orchestratorId + ",orcVersionId:" + orcVersionId); + try { + res = orchestratorContext.getDSSOrchestratorPlugin(ExportDSSOrchestratorPlugin.class).exportOrchestrator(userName, + workspace.getWorkspaceName(), + orchestratorId, + orcVersionId, + projectName, + dssLabelList, + addOrcVersion, + workspace); + } catch (Exception e) { + logger.error("export orchestrator failed for ", e); + throw new DSSErrorException(100789, "export orchestrator failed for " + e.getMessage()); + } + if (null != res) { + Map downRes = bmlService.download(userName, + res.get("resourceId").toString(), + res.get("version").toString()); + + InputStream inputStream = (InputStream) downRes.get("is"); + try { + IOUtils.copy(inputStream, resp.getOutputStream()); + resp.getOutputStream().flush(); + } finally { + IOUtils.closeQuietly(inputStream); + } + } + } + + //生成label list + public List getDSSLabelList(String labels) { + //原来的逻辑 +// List dssLabelList = Arrays.asList(dssLabels.split(",")).stream().map(label -> { +// DSSLabel dssLabel = new EnvDSSLabel(label); +// return dssLabel; +// }).collect(Collectors.toList()); + String labelStr = DSSCommonUtils.ENV_LABEL_VALUE_DEV; + Map labelMap = DSSCommonUtils.COMMON_GSON.fromJson(labels, Map.class); + if (labelMap.containsKey(LabelKeyConvertor.ROUTE_LABEL_KEY)) { + labelStr = (String) labelMap.get(LabelKeyConvertor.ROUTE_LABEL_KEY); + } + List dssLabelList = Arrays.asList(new EnvDSSLabel(labelStr)); + return dssLabelList; + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/restful/OrchestratorRestful.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/restful/OrchestratorRestful.java new file mode 100644 index 000000000..8e3cbed25 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/restful/OrchestratorRestful.java @@ -0,0 +1,150 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.restful; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.DSSLabelUtil; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.label.LabelKeyConvertor; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; +import com.webank.wedatasphere.dss.orchestrator.server.entity.query.QueryOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.server.service.OrchestratorService; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.codehaus.jackson.JsonNode; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.validation.Valid; +import javax.ws.rs.Consumes; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.Arrays; +import java.util.List; +import java.util.Map; + + +@Component +@Path("/dss/framework/orchestrator") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class OrchestratorRestful { + private final static Logger LOGGER = LoggerFactory.getLogger(OrchestratorRestful.class); + @Autowired + OrchestratorService orchestratorService; + + @POST + @Path("/addOrchestrator") + public Response addOrchestrator(@Context HttpServletRequest req, JsonNode json) throws Exception { + String userName = SecurityFilter.getLoginUsername(req); + String name = json.get("name").getTextValue(); + String workspaceName = json.get("workspaceName").getTextValue(); + String projectName = json.get("projectName").getTextValue(); + String typeStr = json.get("type").getTextValue(); + String desc = json.get("desc").getTextValue(); + Long projectID = json.get("projectID").getLongValue(); +// List dssLabelList = Arrays.asList(dssLabels.split(",")).stream().map(label -> { +// DSSLabel dssLabel = new EnvDSSLabel(label); +// return dssLabel; +// }).collect(Collectors.toList()); + List dssLabelList = getDSSLabelList(json); + DSSOrchestratorInfo dssOrchestratorInfo = new DSSOrchestratorInfo(); + dssOrchestratorInfo.setName(name); + dssOrchestratorInfo.setCreator(userName); + dssOrchestratorInfo.setDesc(desc); + dssOrchestratorInfo.setType(typeStr); + OrchestratorVo orchestratorVo = orchestratorService.createOrchestrator(userName, + workspaceName, + projectName, + projectID, + desc, + dssOrchestratorInfo, + dssLabelList); + return Message.messageToResponse(Message.ok().data("OrchestratorVo", orchestratorVo)); + } + + + @POST + @Path("/rollbackOrchestrator") + public Response rollbackOrchestrator(@Context HttpServletRequest request, JsonNode jsonNode) { + String username = SecurityFilter.getLoginUsername(request); + Long orchestratorId = jsonNode.get("orchestratorId").getLongValue(); + String version = jsonNode.get("version").getTextValue(); + Long projectId = jsonNode.get("projectId").getLongValue(); + String projectName = jsonNode.get("projectName").getTextValue(); + Workspace workspace = SSOHelper.getWorkspace(request); + DSSLabel envDSSLabel = getDSSLabelList(jsonNode).get(0); + try { + String newVersion = orchestratorService.rollbackOrchestrator(username, projectId, projectName, orchestratorId, version, envDSSLabel, workspace); + Message message = Message.ok("回滚版本成功").data("newVersion", newVersion); + return Message.messageToResponse(message); + } catch (final Throwable t) { + LOGGER.error("Failed to rollback orchestrator for user {} orchestratorId {}, projectId {} version {}", + username, orchestratorId, projectId, version, t); + return Message.messageToResponse(Message.error("回滚工作流版本失败")); + } + } + + @POST + @Path("/openOrchestrator") + public Response openOrchestrator(@Context HttpServletRequest req, JsonNode json) throws Exception { + String openUrl = ""; + String userName = SecurityFilter.getLoginUsername(req); + List dssLabelList = getDSSLabelList(json); + String workspaceName = json.get("workspaceName").getTextValue(); + Long orchestratorId = json.get("orchestratorId").getLongValue(); + openUrl = orchestratorService.openOrchestrator(userName, workspaceName, orchestratorId, dssLabelList); + OrchestratorVo orchestratorVo = orchestratorService.getOrchestratorVoById(orchestratorId); + LOGGER.info("open url is {}, orcId is {}, dssLabels is {}", openUrl, orchestratorId, dssLabelList); + return Message.messageToResponse(Message.ok(). + data("OrchestratorOpenUrl", openUrl). + data("OrchestratorVo", orchestratorVo)); + } + + /** + * 获取编排模式下的所有版本号 + * + * @param queryOrchestratorVersion + * @return + * @throws Exception + */ + @POST + @Path("/getVersionByOrchestratorId") + public Response getVersionByOrchestratorId(@Valid QueryOrchestratorVersion queryOrchestratorVersion) throws Exception { + List list = orchestratorService.getVersionByOrchestratorId(queryOrchestratorVersion.getOrchestratorId()); + return Message.messageToResponse(Message.ok().data("list", list)); + } + + //生成label list + public List getDSSLabelList(JsonNode json) { + JsonNode labelJsonNode = json.get(DSSCommonUtils.DSS_LABELS_KEY); + String dssLabel = labelJsonNode.get(LabelKeyConvertor.ROUTE_LABEL_KEY).getTextValue(); + List dssLabelList = Arrays.asList(new EnvDSSLabel(dssLabel)); + return dssLabelList; + } +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/OrchestratorPluginService.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/OrchestratorPluginService.java new file mode 100644 index 000000000..f32984959 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/OrchestratorPluginService.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.service; + +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestFrameworkConvertOrchestration; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseConvertOrchestrator; + + +public interface OrchestratorPluginService { + + ResponseConvertOrchestrator convertOrchestration(RequestFrameworkConvertOrchestration requestConversionOrchestration); + + ResponseConvertOrchestrator getConvertOrchestrationStatus(String id); + +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/OrchestratorService.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/OrchestratorService.java new file mode 100644 index 000000000..beefda5eb --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/OrchestratorService.java @@ -0,0 +1,99 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.service; + + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import java.util.List; + + +public interface OrchestratorService { + /** + * 新建编排，实例化一个用户的Orchestrator,并创建数据库记录 + * + * @param dssOrchestratorInfo + * @return + */ + OrchestratorVo createOrchestrator(String userName, + String workspaceName, + String projectName, + Long projectId, + String description, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) throws Exception; + + + /** + * 更新编排，更新编排的基本信息 + * + * @param dssOrchestratorInfo + */ + void updateOrchestrator(String userName, + String workspaceName, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) throws Exception; + + /** + * 删除编排，根据编排ID删除一个编排 + * + * @param orchestratorInfoId + */ + void deleteOrchestrator(String userName, + String workspaceName, + String projectName, + Long orchestratorInfoId, + List dssLabels) throws Exception; + + + /** + * 返回一个编排，包含编排的基本信息和最新版本信息 + * + * @param orchestratorId + * @return + */ + OrchestratorVo getOrchestratorVoById(Long orchestratorId); + + /** + * 根据一个集合查找 + * + * @param orchestratorIds + * @return + */ + + List getOrchestratorVoList(List orchestratorIds); + + String openOrchestrator(String userName, String workspaceName, Long orchestratorId, List dssLabels) throws Exception; + /** + * 获取编排模式下的版本号 + * + * @param orchestratorId + * @return + */ + List getVersionByOrchestratorId(Long orchestratorId); + +// OrchestratorInfo getOrchestratorInfo(String username, Long workflowId); + + List getOrchestratorVersions(String username, Long projectId, Long orchestratorId); + + String rollbackOrchestrator(String username, Long projectId, String projectName, + Long orchestratorId, String version, DSSLabel dssLabel, Workspace workspace) throws Exception; + +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/impl/OrchestratorPluginServiceImpl.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/impl/OrchestratorPluginServiceImpl.java new file mode 100644 index 000000000..b00762ef3 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/impl/OrchestratorPluginServiceImpl.java @@ -0,0 +1,160 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.service.impl; + +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestFrameworkConvertOrchestration; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseConvertOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext; +import com.webank.wedatasphere.dss.orchestrator.db.dao.OrchestratorMapper; +import com.webank.wedatasphere.dss.orchestrator.publish.ExportDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.publish.job.ConversionJobEntity; +import com.webank.wedatasphere.dss.orchestrator.publish.job.OrchestratorConversionJob; +import com.webank.wedatasphere.dss.orchestrator.server.service.OrchestratorPluginService; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import com.webank.wedatasphere.linkis.manager.label.builder.factory.LabelBuilderFactoryContext; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.atomic.AtomicInteger; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.transaction.annotation.Transactional; + + +public class OrchestratorPluginServiceImpl implements OrchestratorPluginService { + + private final Logger LOGGER = LoggerFactory.getLogger(getClass()); + + @Autowired + private OrchestratorMapper orchestratorMapper; + + @Autowired + private DSSOrchestratorContext dssOrchestratorContext; + + private ExecutorService releaseThreadPool = Utils.newCachedThreadPool(50, "Convert-Orchestration-Thread-", true); + private Map orchestratorConversionJobMap = new ConcurrentHashMap<>(); + private AtomicInteger idGenerator = new AtomicInteger(); + + @Override + public ResponseConvertOrchestrator convertOrchestration(RequestFrameworkConvertOrchestration requestConversionOrchestration) { + Long toPublishOrcId; + if(requestConversionOrchestration.getOrcAppId() != null) { + OrchestratorInfo orchestratorInfo = orchestratorMapper.getOrcInfoByAppId(requestConversionOrchestration.getOrcAppId()); + toPublishOrcId = orchestratorInfo.getOrchestratorId(); + } else if(requestConversionOrchestration.getOrcIds() != null && !requestConversionOrchestration.getOrcIds().isEmpty()) { + toPublishOrcId = requestConversionOrchestration.getOrcIds().get(0); + } else { + return new ResponseConvertOrchestrator("-1", ResponseOperateOrchestrator.failed("Both orcAppId and orcIds are not exists.")); + } + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(toPublishOrcId); + long projectId = dssOrchestratorInfo.getProjectId(); + //把orcId装车第三方的AppId + List refAppIdList = new ArrayList<>(); + List publishedOrcIds; + List labels = LabelBuilderFactoryContext.getLabelBuilderFactory().getLabels(requestConversionOrchestration.getLabels()); + if(requestConversionOrchestration.isConvertAllOrcs()) { + //这个地方应该是要获取所有的已经发布过的orchestrator + publishedOrcIds = orchestratorMapper.getAllOrcIdsByProjectId(projectId); + if (!publishedOrcIds.contains(toPublishOrcId)){ + publishedOrcIds.add(toPublishOrcId); + } + for (Long orcId : publishedOrcIds) { + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(orcId); + refAppIdList.add(dssOrchestratorVersion.getAppId()); + } + } else { + publishedOrcIds = new ArrayList<>(); + if(requestConversionOrchestration.getOrcAppId() != null) { + publishedOrcIds.add(toPublishOrcId); + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(toPublishOrcId); + refAppIdList.add(dssOrchestratorVersion.getAppId()); + } + if(requestConversionOrchestration.getOrcIds() != null && !requestConversionOrchestration.getOrcIds().isEmpty()) { + for (Long orcId : requestConversionOrchestration.getOrcIds()) { + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(orcId); + publishedOrcIds.add(orcId); + refAppIdList.add(dssOrchestratorVersion.getAppId()); + } + } + } + OrchestratorConversionJob job = new OrchestratorConversionJob(); + job.setId(generateId()); + ConversionJobEntity entity = new ConversionJobEntity(); + entity.setResponse(ResponseOperateOrchestrator.inited()); + entity.setCreateTime(new Date()); + entity.setUserName(requestConversionOrchestration.getUserName()); + entity.setOrcIdList(publishedOrcIds); + entity.setRefAppIdList(refAppIdList); + entity.setLabels(labels); + entity.setWorkspace((Workspace) requestConversionOrchestration.getWorkspace()); + DSSProject dssProject = new DSSProject(); + dssProject.setId(projectId); + entity.setProject(dssProject); + job.setConversionJobEntity(entity); + job.setConversionDSSOrchestratorPlugins(dssOrchestratorContext.getOrchestratorPlugins()); + job.afterConversion(response -> this.updateDBAfterConversion(response, job, requestConversionOrchestration)); + //submit it + releaseThreadPool.submit(job); + orchestratorConversionJobMap.put(job.getId(), job); + return new ResponseConvertOrchestrator(job.getId(), entity.getResponse()); + } + + @Override + public ResponseConvertOrchestrator getConvertOrchestrationStatus(String id) { + OrchestratorConversionJob job = orchestratorConversionJobMap.get(id); + if(job.getConversionJobEntity().getResponse().isCompleted()) { + orchestratorConversionJobMap.remove(id); + } + return new ResponseConvertOrchestrator(job.getId(), job.getConversionJobEntity().getResponse()); + } + + private String generateId() { + return String.valueOf(idGenerator.getAndIncrement()); + } + + @Transactional(rollbackFor = Exception.class) + private void updateDBAfterConversion(ResponseOperateOrchestrator response, + OrchestratorConversionJob job, + RequestFrameworkConvertOrchestration requestConversionOrchestration) { + LOGGER.info("{} completed with status {}.", job.getId(), response.getJobStatus()); + if(response.isSucceed()) { + //1. 进行导出,用于升级版本,目的是为了复用原来的代码 + List orcIdList = job.getConversionJobEntity().getOrcIdList(); + orcIdList.forEach(DSSExceptionUtils.handling(orcId -> { + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(orcId); + dssOrchestratorContext.getDSSOrchestratorPlugin(ExportDSSOrchestratorPlugin.class) + .orchestratorVersionIncrease(orcId, job.getConversionJobEntity().getUserName(), requestConversionOrchestration.getComment(), + requestConversionOrchestration.getWorkspace().getWorkspaceName(), dssOrchestratorInfo, + job.getConversionJobEntity().getProject().getName(), job.getConversionJobEntity().getLabels()); + })); + //2. 做一个标记表示已经发布过了 + orcIdList.forEach(orchestratorMapper::setPublished); + } + } +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/impl/OrchestratorServiceImpl.java b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/impl/OrchestratorServiceImpl.java new file mode 100644 index 000000000..d9e37e1de --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/java/com/webank/wedatasphere/dss/orchestrator/server/service/impl/OrchestratorServiceImpl.java @@ -0,0 +1,376 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.service.impl; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.exception.DSSOrchestratorErrorException; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCopyRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCopyResponseRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorDeleteRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorOpenRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorUpdateRef; +import com.webank.wedatasphere.dss.orchestrator.core.utils.OrchestratorUtils; +import com.webank.wedatasphere.dss.orchestrator.db.dao.OrchestratorMapper; +import com.webank.wedatasphere.dss.orchestrator.loader.OrchestratorManager; +import com.webank.wedatasphere.dss.orchestrator.loader.utils.OrchestratorLoaderUtils; +import com.webank.wedatasphere.dss.orchestrator.server.service.OrchestratorService; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.UrlResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.service.RefCRUDService; +import com.webank.wedatasphere.dss.standard.app.development.service.RefQueryService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AppConnRefFactoryUtils; +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.UUID; +import javafx.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + + + +@Service +public class OrchestratorServiceImpl implements OrchestratorService { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorServiceImpl.class); + @Autowired + private OrchestratorManager orchestratorManager; + @Autowired + private OrchestratorMapper orchestratorMapper; + @Autowired + private ContextService contextService; + + @Override + @Transactional(rollbackFor = Exception.class) + public OrchestratorVo createOrchestrator(String userName, + String workspaceName, + String projectName, + Long projectId, + String description, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) throws Exception { + OrchestratorVo orchestratorVo = new OrchestratorVo(); + //todo 增加校验 + String uuid = UUID.randomUUID().toString(); + + //作为Orchestrator的唯一标识，包括跨环境导入导出也不发生变化。 + dssOrchestratorInfo.setUUID(uuid); + orchestratorMapper.addOrchestrator(dssOrchestratorInfo); + Pair standMap = OrchestratorLoaderUtils.getOrcDevelopStandard(userName, workspaceName, dssOrchestratorInfo, dssLabels); + DSSOrchestrator dssOrchestrator = orchestratorManager.getOrCreateOrchestrator(userName, + workspaceName, dssOrchestratorInfo.getType(), dssOrchestratorInfo.getAppConnName(), dssLabels); + AppConn orchestratorAppConn = dssOrchestrator.getAppConn(); + if (null != standMap && null != orchestratorAppConn) { + //访问工作流微模块创建工作流 + OrchestratorCreateRequestRef ref = AppConnRefFactoryUtils.newAppConnRef(OrchestratorCreateRequestRef.class, + orchestratorAppConn.getClass().getClassLoader(), dssOrchestratorInfo.getType()); + if (null != ref) { + ref.setUserName(userName); + ref.setWorkspaceName(workspaceName); + ref.setProjectName(projectName); + ref.setProjectId(projectId); + ref.setDssOrchestratorInfo(dssOrchestratorInfo); + String version = OrchestratorUtils.generateNewVersion(); + String contextId = contextService.createContextID(workspaceName, projectName, dssOrchestratorInfo.getName(), version, userName); + ref.setContextID(contextId); + LOGGER.info("Create a new ContextId: {} ", contextId); + RefCRUDService crudService =standMap.getValue().getRefCRUDService(standMap.getKey()); + if (crudService != null) { + CommonResponseRef appRef = (CommonResponseRef) crudService + .getRefCreationOperation().createRef(ref); + if (null != appRef) { + DSSOrchestratorVersion dssOrchestratorVersion = new DSSOrchestratorVersion(); + dssOrchestratorVersion.setOrchestratorId(dssOrchestratorInfo.getId()); + dssOrchestratorVersion.setAppId(appRef.getOrcId()); + dssOrchestratorVersion.setContent(appRef.getContent()); + dssOrchestratorVersion.setComment(description); + dssOrchestratorVersion.setProjectId(projectId); + dssOrchestratorVersion.setSource("Orchestrator create"); + dssOrchestratorVersion.setUpdater(userName); + dssOrchestratorVersion.setVersion(version); + dssOrchestratorVersion.setUpdateTime(new Date()); + dssOrchestratorVersion.setFormatContextId(contextId); + orchestratorMapper.addOrchestratorVersion(dssOrchestratorVersion); + orchestratorVo.setDssOrchestratorInfo(dssOrchestratorInfo); + orchestratorVo.setDssOrchestratorVersion(dssOrchestratorVersion); + } else { + LOGGER.error("create ref return appRef is null"); + throw new DSSErrorException(100068, "Use appInstance create workflow failed!"); + } + } else { + LOGGER.error("crudService is null can not continue"); + throw new DSSErrorException(60092, "create crudService is null"); + } + } else { + LOGGER.error("ref is null can not continue"); + throw new DSSErrorException(60093, "ref is null can not continue"); + } + } else { + throw new DSSErrorException(100069, "Can not find correct devCRUDService"); + } + return orchestratorVo; + } + + @Override + public void updateOrchestrator(String userName, + String workspaceName, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) throws Exception { + orchestratorMapper.updateOrchestrator(dssOrchestratorInfo); + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(dssOrchestratorInfo.getId()); + + //todo 目前写死为工作流，这里应该是要和dssOrchestratorInfo.type作为参数传入 + AppConn appConn = AppConnManager.getAppConnManager().getAppConn("workflow"); + if(appConn==null){ + LOGGER.error("appConn is null,not update workflow"); + return; + } + OrchestratorUpdateRef orchestratorUpdateRef = AppConnRefFactoryUtils.newAppConnRef(OrchestratorUpdateRef.class, + appConn.getClass().getClassLoader(), dssOrchestratorInfo.getType()); + if (orchestratorUpdateRef != null) { + orchestratorUpdateRef.setOrcId(dssOrchestratorVersion.getAppId()); + orchestratorUpdateRef.setUserName(userName); + orchestratorUpdateRef.setDescription(dssOrchestratorInfo.getComment()); + orchestratorUpdateRef.setOrcName(dssOrchestratorInfo.getName()); + orchestratorUpdateRef.setUses(dssOrchestratorInfo.getUses()); + //update ref orchestrator info + Pair standMap = OrchestratorLoaderUtils.getOrcDevelopStandard(userName, workspaceName, dssOrchestratorInfo, dssLabels); + if (null != standMap ) { + RefCRUDService crudService = standMap.getValue().getRefCRUDService(standMap.getKey()); + if (null != crudService) { + crudService.getRefUpdateOperation().updateRef(orchestratorUpdateRef); + } + } else { + LOGGER.error("processService is null,can not do crud service"); + DSSExceptionUtils.dealErrorException(60056, "process service is null, can not do crud", DSSErrorException.class); + } + + } else { + LOGGER.error("update ref is null, can not continue doing"); + DSSExceptionUtils.dealErrorException(61123, "update ref is null, can not continue doing", DSSErrorException.class); + } + + } + + @Override + public void deleteOrchestrator(String userName, + String workspaceName, + String projectName, + Long orchestratorInfoId, + List dssLabels) throws Exception { + + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(orchestratorInfoId); + if(null== dssOrchestratorInfo){ + LOGGER.error("dssOrchestratorInfo is null,no need to delete"); + return; + } + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(orchestratorInfoId); + + //todo 是否需要删除版本信息 + + //todo 删除版本信息对应的工作流信息 + + OrchestratorDeleteRequestRef orchestratorDeleteRequestRef = null; + try { + //todo 目前写死为工作流，这里应该是要和dssOrchestratorInfo.type作为参数传入 + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(dssOrchestratorInfo.getType()); + if(appConn == null){ + LOGGER.error("appConn is null, not delete {}." + dssOrchestratorInfo.getType()); + return; + } + orchestratorDeleteRequestRef = AppConnRefFactoryUtils.newAppConnRefByPackageName(OrchestratorDeleteRequestRef.class, + appConn.getClass().getClassLoader(), appConn.getClass().getPackage().getName()); + } catch (Exception e) { + LOGGER.error("Failed to create a new ref for {}.", OrchestratorDeleteRequestRef.class, e); + } + assert orchestratorDeleteRequestRef != null; + Pair standMap = OrchestratorLoaderUtils.getOrcDevelopStandard(userName, workspaceName, dssOrchestratorInfo, dssLabels); + + RefCRUDService refCRUDService = standMap.getValue().getRefCRUDService (standMap.getKey()); + + //删除只需要 + orchestratorDeleteRequestRef.setAppId(dssOrchestratorVersion.getAppId()); + orchestratorDeleteRequestRef.setOrcId(orchestratorInfoId); + orchestratorDeleteRequestRef.setUserName(userName); + orchestratorDeleteRequestRef.setDSSLabels(dssLabels); + if (null != refCRUDService) { + refCRUDService.getRefDeletionOperation().deleteRef(orchestratorDeleteRequestRef); + } + + orchestratorMapper.deleteOrchestrator(orchestratorInfoId); + } + + @Override + public List getOrchestratorVoList(List orchestratorIds) { + List orchestratorVoList = new ArrayList<>(); + orchestratorIds.stream().forEach(orchestratorId -> { + OrchestratorVo orchestratorVo = getOrchestratorVoById(orchestratorId); + orchestratorVoList.add(orchestratorVo); + }); + return orchestratorVoList; + } + + @Override + public String openOrchestrator(String userName, String workspaceName, Long orchestratorId, List dssLabels) throws Exception { + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(orchestratorId); + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(orchestratorId); + if (null == dssOrchestratorInfo || null == dssOrchestratorVersion) { + throw new DSSOrchestratorErrorException(1000856, "can not find orc from db for orcId: " + orchestratorId); + } + OrchestratorOpenRequestRef orchestratorOpenRequestRef = null; + Pair standMap = OrchestratorLoaderUtils.getOrcDevelopStandard(userName, workspaceName, dssOrchestratorInfo, dssLabels); + + DSSOrchestrator dssOrchestrator = orchestratorManager.getOrCreateOrchestrator(userName, + workspaceName, dssOrchestratorInfo.getType(), dssOrchestratorInfo.getAppConnName(), dssLabels); + AppConn orchestratorAppConn = dssOrchestrator.getAppConn(); + + try { + orchestratorOpenRequestRef = AppConnRefFactoryUtils.newAppConnRefByPackageName(OrchestratorOpenRequestRef.class, + orchestratorAppConn.getClass().getClassLoader(), orchestratorAppConn.getClass().getPackage().getName()); + } catch (final Exception e) { + LOGGER.error("Failed to open a new ref for {}", OrchestratorOpenRequestRef.class, e); + } + assert orchestratorOpenRequestRef != null; + RefQueryService refQueryService =standMap.getValue().getRefQueryService (standMap.getKey()); + + orchestratorOpenRequestRef.setRefAppId(dssOrchestratorVersion.getAppId()); + orchestratorOpenRequestRef.setOrchestratorId(orchestratorId); + orchestratorOpenRequestRef.setUserName(userName); + orchestratorOpenRequestRef.setSecondaryType(dssOrchestratorInfo.getSecondaryType()); + orchestratorOpenRequestRef.setDSSLabels(dssLabels); + if (null != refQueryService) { + UrlResponseRef urlResponseRef=(UrlResponseRef) refQueryService.getRefQueryOperation().query(orchestratorOpenRequestRef); + return urlResponseRef.getUrl(); + } + return null; + } + + @Override + public OrchestratorVo getOrchestratorVoById(Long orchestratorId) { + + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(orchestratorId); + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(orchestratorId); + + OrchestratorVo orchestratorVo = new OrchestratorVo(); + orchestratorVo.setDssOrchestratorInfo(dssOrchestratorInfo); + orchestratorVo.setDssOrchestratorVersion(dssOrchestratorVersion); + return orchestratorVo; + } + + @Override + public List getVersionByOrchestratorId(Long orchestratorId) { + return orchestratorMapper.getVersionByOrchestratorId(orchestratorId); + } + +// @Override +// public OrchestratorInfo getOrchestratorInfo(String username, Long workflowId) { +// LOGGER.info("{} ask the orcInfo for workflowId {}", username, workflowId); +// OrchestratorInfo orchestratorInfo = orchestratorMapper.getOrcInfoByAppId(workflowId); +// LOGGER.info("workflowId is {} , orcId is {}, orcVersionId is {}", workflowId, orchestratorInfo.getOrchestratorId(), orchestratorInfo.getOrchestratorVersionId()); +// return orchestratorInfo; +// } + + @Override + public List getOrchestratorVersions(String username, Long projectId, Long orchestratorId) { + LOGGER.info("user {} wants to get orc versions in projectId {} for orcId {}", username, projectId, orchestratorId); + List orchestratorVersions = orchestratorMapper.getOrchestratorVersions(projectId, orchestratorId); + LOGGER.info("projectId is {} , orcId is {}, orcVersions are {}", projectId, orchestratorId, orchestratorVersions); + return orchestratorVersions; + } + + + @Override + @Transactional(rollbackFor = Exception.class) + public String rollbackOrchestrator(String userName, Long projectId, String projectName, + Long orchestratorId, String version, DSSLabel dssLabel, Workspace workspace) throws Exception { + //1.新建一个版本 + //2.然后将version的版本内容进行去workflow进行cp + //3.然后把生产的内容进行update到数据库 + String latestVersion = orchestratorMapper.getLatestVersion(orchestratorId); + List labels = new ArrayList<>(); + labels.add(dssLabel); + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(orchestratorId); + String newVersion = OrchestratorUtils.increaseVersion(latestVersion); + DSSOrchestratorVersion dssOrchestratorVersion = new DSSOrchestratorVersion(); + dssOrchestratorVersion.setId(orchestratorId); + dssOrchestratorVersion.setVersion(newVersion); + dssOrchestratorVersion.setUpdateTime(new Date()); + dssOrchestratorVersion.setProjectId(projectId); + dssOrchestratorVersion.setUpdater(userName); + dssOrchestratorVersion.setComment("回滚工作流到版本:" + version); + dssOrchestratorVersion.setSource("rollback from version :" + version); + Long appId = orchestratorMapper.getAppIdByVersion(orchestratorId, version); + + Pair standMap = OrchestratorLoaderUtils.getOrcDevelopStandard(userName, workspace.getWorkspaceName(), dssOrchestratorInfo, labels); + + if(standMap == null){ + LOGGER.error("dev stand Service is null"); + throw new DSSErrorException(61105, "dev stand Service is null"); + } + RefCRUDService refcrudservice = standMap.getValue().getRefCRUDService(standMap.getKey()); + if (null != refcrudservice) { + try { + OrchestratorCopyRequestRef orchestratorCopyRequestRef = + AppConnRefFactoryUtils.newAppConnRef(OrchestratorCopyRequestRef.class, + refcrudservice.getClass().getClassLoader(), dssOrchestratorInfo.getType()); + orchestratorCopyRequestRef.setCopyOrcAppId(appId); + orchestratorCopyRequestRef.setCopyOrcVersionId(dssOrchestratorVersion.getOrchestratorId()); + orchestratorCopyRequestRef.setUserName(userName); + Field field = orchestratorCopyRequestRef.getClass().getDeclaredField("projectName"); + field.setAccessible(true); + field.set(orchestratorCopyRequestRef, projectName); + + //5、生成上下文ContextId + String contextId = contextService.createContextID(workspace.getWorkspaceName(), projectName, dssOrchestratorInfo.getName(), dssOrchestratorVersion.getVersion(), userName); + dssOrchestratorVersion.setContextId(contextId); + LOGGER.info("Create a new ContextId for import: {} ", contextId); + + orchestratorCopyRequestRef.setContextID(contextId); + OrchestratorCopyResponseRef orchestratorCopyResponseRef = + (OrchestratorCopyResponseRef) refcrudservice.getRefCopyOperation().copyRef(orchestratorCopyRequestRef); + dssOrchestratorVersion.setAppId(orchestratorCopyResponseRef.getCopyTargetAppId()); + dssOrchestratorVersion.setContent(orchestratorCopyResponseRef.getCopyTargetContent()); + dssOrchestratorVersion.setFormatContextId(contextId); + //update appConn node contextId + orchestratorMapper.addOrchestratorVersion(dssOrchestratorVersion); + } catch (final Throwable t) { + LOGGER.error("Faild to copy app in orchestrator server", t); + DSSExceptionUtils.dealErrorException(60099, "Faild to copy app in orchestrator server", t, DSSOrchestratorErrorException.class); + } + return dssOrchestratorVersion.getVersion(); + } else { + throw new DSSOrchestratorErrorException(10023, "获取第三方应用的Ref为空，不能完成拷贝操作！"); + } + } + +} diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/DSSOrchestratorServerApplication.scala b/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/DSSOrchestratorServerApplication.scala new file mode 100644 index 000000000..848f1ea22 --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/DSSOrchestratorServerApplication.scala @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server + +import com.webank.wedatasphere.dss.common.utils.DSSMainHelper +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} + + +object DSSOrchestratorServerApplication extends Logging { + + val userName: String = System.getProperty("user.name") + val hostName: String = Utils.getComputerName + + def main(args: Array[String]): Unit = { + val serviceName = System.getProperty("serviceName")//ProjectConf.SERVICE_NAME.getValue + DSSMainHelper.formatPropertyFiles(serviceName) + val allArgs = args ++ DSSMainHelper.getExtraSpringOptions + System.setProperty("hostName", hostName) + System.setProperty("userName", userName) + info(s"Ready to start $serviceName with args: ${allArgs.toList}.") + println(s"Test Ready to start $serviceName with args: ${allArgs.toList}.") + DataWorkCloudApplication.main(allArgs) + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/receiver/DSSOrchestratorChooser.scala b/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/receiver/DSSOrchestratorChooser.scala new file mode 100644 index 000000000..c5ccda43c --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/receiver/DSSOrchestratorChooser.scala @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.receiver + +import com.webank.wedatasphere.dss.orchestrator.common.protocol._ +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext +import com.webank.wedatasphere.dss.orchestrator.server.service.{OrchestratorPluginService, OrchestratorService} +import com.webank.wedatasphere.linkis.rpc.{RPCMessageEvent, Receiver, ReceiverChooser} +import javax.annotation.PostConstruct +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.stereotype.Component + + + +@Component +class DSSOrchestratorChooser extends ReceiverChooser { + + @Autowired + var orchestratorService: OrchestratorService = _ + + @Autowired + var orchestratorPluginService: OrchestratorPluginService = _ + + @Autowired + var orchestratorContext: DSSOrchestratorContext = _ + + var receiver: Option[DSSOrchestratorReceiver] = _ + + @PostConstruct + def init(): Unit = receiver = Some(new DSSOrchestratorReceiver(orchestratorService,orchestratorPluginService,orchestratorContext)) + + override def chooseReceiver(event: RPCMessageEvent): Option[Receiver] = event.message match { + case _: RequestCreateOrchestrator => receiver + case _: RequestUpdateOrchestrator => receiver + case _: RequestDeleteOrchestrator => receiver + case _: RequestExportOrchestrator => receiver + case _: RequestImportOrchestrator => receiver + case _: RequestQueryOrchestrator => receiver + case _: RequestFrameworkConvertOrchestration => receiver + case _: RequestFrameworkConvertOrchestrationStatus => receiver + case _: RequestOrchestratorVersion => receiver + case _ => None + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/receiver/DSSOrchestratorReceiver.scala b/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/receiver/DSSOrchestratorReceiver.scala new file mode 100644 index 000000000..946b0481b --- /dev/null +++ b/dss-framework/dss-framework-orchestrator-server/src/main/scala/com/webank/wedatasphere/dss/orchestrator/server/receiver/DSSOrchestratorReceiver.scala @@ -0,0 +1,114 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.server.receiver + +import java.util + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException +import com.webank.wedatasphere.dss.common.protocol._ +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo +import com.webank.wedatasphere.dss.orchestrator.common.protocol._ +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext +import com.webank.wedatasphere.dss.orchestrator.publish.{ExportDSSOrchestratorPlugin, ImportDSSOrchestratorPlugin} +import com.webank.wedatasphere.dss.orchestrator.server.service.{OrchestratorPluginService, OrchestratorService} +import com.webank.wedatasphere.dss.standard.app.sso.Workspace +import com.webank.wedatasphere.linkis.rpc.{Receiver, Sender} + +import scala.concurrent.duration.Duration + + + +class DSSOrchestratorReceiver(orchestratorService: OrchestratorService,orchestratorPluginService: OrchestratorPluginService,orchestratorContext: DSSOrchestratorContext) extends Receiver { + + override def receive(message: Any, sender: Sender): Unit = {} + + override def receiveAndReply(message: Any, sender: Sender): Any = message match { + case reqCreateOrchestrator: RequestCreateOrchestrator => + val orchestratorVo: OrchestratorVo = orchestratorService.createOrchestrator(reqCreateOrchestrator.getUserName, + reqCreateOrchestrator.getWorkspaceName, + reqCreateOrchestrator.getProjectName, + reqCreateOrchestrator.getProjectId, + reqCreateOrchestrator.getDescription, + reqCreateOrchestrator.getDssOrchestratorInfo, + reqCreateOrchestrator.getDssLabels) + ResponseCreateOrchestrator(orchestratorVo.getDssOrchestratorInfo.getId, orchestratorVo.getDssOrchestratorVersion.getId) + + case reqUpdateOrchestrator: RequestUpdateOrchestrator => + orchestratorService.updateOrchestrator(reqUpdateOrchestrator.getUserName, + reqUpdateOrchestrator.getWorkspaceName, + reqUpdateOrchestrator.getDssOrchestratorInfo, + reqUpdateOrchestrator.getDssLabels) + ResponseOperateOrchestrator.success() + + case reqDeleteOrchestrator: RequestDeleteOrchestrator => + orchestratorService.deleteOrchestrator(reqDeleteOrchestrator.getUserName, + reqDeleteOrchestrator.getWorkspaceName, + reqDeleteOrchestrator.getProjectName, + reqDeleteOrchestrator.getOrchestratorId, + reqDeleteOrchestrator.getDssLabels + ) + ResponseOperateOrchestrator.success() + + case reqExportOrchestrator: RequestExportOrchestrator => + val dssExportOrcResource: util.Map[String, AnyRef] = orchestratorContext.getDSSOrchestratorPlugin(classOf[ExportDSSOrchestratorPlugin]).exportOrchestrator( + reqExportOrchestrator.getUserName, + reqExportOrchestrator.getWorkspaceName, + reqExportOrchestrator.getOrchestratorId, + reqExportOrchestrator.getOrcVersionId, + reqExportOrchestrator.getProjectName, + reqExportOrchestrator.getDssLabels, + reqExportOrchestrator.getAddOrcVersion, + DSSCommonUtils.COMMON_GSON.fromJson(reqExportOrchestrator.getWorkspaceStr, classOf[Workspace])) + ResponseExportOrchestrator(dssExportOrcResource.get("resourceId").toString, + dssExportOrcResource.get("version").toString, dssExportOrcResource.get("orcVersionId").asInstanceOf[Long] + ) + + case requestImportOrchestrator: RequestImportOrchestrator => + val importOrcId = orchestratorContext.getDSSOrchestratorPlugin(classOf[ImportDSSOrchestratorPlugin]).importOrchestrator( + requestImportOrchestrator.getUserName, + requestImportOrchestrator.getWorkspaceName, + requestImportOrchestrator.getProjectName, + requestImportOrchestrator.getProjectId, + requestImportOrchestrator.getResourceId, + requestImportOrchestrator.getBmlVersion, + requestImportOrchestrator.getDssLabels, + DSSCommonUtils.COMMON_GSON.fromJson(requestImportOrchestrator.getWorkspaceStr, classOf[Workspace])) + ResponseImportOrchestrator(importOrcId) + + case requestQueryOrchestrator: RequestQueryOrchestrator => + val requestIdList = requestQueryOrchestrator.getOrchestratorIds + val queryOrchestratorList: util.List[OrchestratorVo] = orchestratorService.getOrchestratorVoList(requestIdList) + new ResponseQueryOrchestrator(queryOrchestratorList) + + case requestConversionOrchestration: RequestFrameworkConvertOrchestration => + //发布调度 + orchestratorPluginService.convertOrchestration(requestConversionOrchestration) + case requestConversionOrchestrationStatus: RequestFrameworkConvertOrchestrationStatus => + orchestratorPluginService.getConvertOrchestrationStatus(requestConversionOrchestrationStatus.getId) + + case requestOrchestratorVersion: RequestOrchestratorVersion => + val projectId = requestOrchestratorVersion.getProjectId + val username = requestOrchestratorVersion.getUsername + val orchestratorId = requestOrchestratorVersion.getOrchestratorId + val orchestratorVersions = orchestratorService.getOrchestratorVersions(username, projectId, orchestratorId) + new ResponseOrchetratorVersion(projectId, orchestratorId, orchestratorVersions) + case _ => throw new DSSErrorException(90000, "") + } + + override def receiveAndReply(message: Any, duration: Duration, sender: Sender): Any = {} +} diff --git a/dss-framework/dss-framework-project-server/pom.xml b/dss-framework/dss-framework-project-server/pom.xml new file mode 100644 index 000000000..d5b79a300 --- /dev/null +++ b/dss-framework/dss-framework-project-server/pom.xml @@ -0,0 +1,252 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + + 4.0.0 + + dss-framework-project-server + + + + com.webank.wedatasphere.dss + dss-framework-common + ${dss.version} + + + commons-math3 + org.apache.commons + + + + + + org.apache.commons + commons-math3 + provided + + + + com.webank.wedatasphere.dss + dss-appconn-framework + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-structure-integration-standard + ${dss.version} + + + commons-collections + commons-collections + + + dss-common + com.webank.wedatasphere.dss + + + linkis-common + com.webank.wedatasphere.linkis + + + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + + + commons-collections + commons-collections + + + linkis-common + com.webank.wedatasphere.linkis + + + commons-lang3 + org.apache.commons + + + commons-logging + commons-logging + + + + + + com.webank.wedatasphere.dss + dss-orchestrator-db + ${dss.version} + + + + + + org.apache.commons + commons-lang3 + ${commons.lang3.version} + provided + + + + org.glassfish.jersey.ext + jersey-bean-validation + 2.21 + provided + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + linkis-common + com.webank.wedatasphere.linkis + + + + + com.webank.wedatasphere.dss + dss-framework-workspace-server + ${dss.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + commons-logging + commons-logging + + + commons-codec + commons-codec + + + commons-lang3 + org.apache.commons + + + json4s-jackson_2.11 + org.json4s + + + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-sender-service + ${dss.version} + provided + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + org.apache.maven.plugins + maven-compiler-plugin + + 8 + 8 + + + + + + src/main/java + + **/*.xml + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-project-server/src/main/assembly/distribution.xml b/dss-framework/dss-framework-project-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..a506dcb06 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/assembly/distribution.xml @@ -0,0 +1,44 @@ + + + + dss-framework-project-server + + dir + + true + dss-framework-project-server + + + + + + lib + true + true + false + true + true + + + + + + + diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ProjectConf.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ProjectConf.java new file mode 100644 index 000000000..67bd65d9d --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ProjectConf.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public interface ProjectConf { + + CommonVars SUPPORT_ABILITY = CommonVars.apply("wds.dss.framework.project.support.ability", "import,export,publish"); + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ProjectSpringConf.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ProjectSpringConf.java new file mode 100644 index 000000000..f29748335 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ProjectSpringConf.java @@ -0,0 +1,43 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.conf; + +import com.webank.wedatasphere.dss.framework.project.service.DSSFrameworkOrchestratorService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectUserService; +import com.webank.wedatasphere.dss.framework.project.service.impl.DSSFrameworkOrchestratorServiceImpl; +import com.webank.wedatasphere.dss.framework.project.service.impl.DSSProjectUserServiceImpl; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + + +@Configuration +public class ProjectSpringConf { + + @Bean + @ConditionalOnMissingBean + public DSSProjectUserService createDSSProjectUserService() { + return new DSSProjectUserServiceImpl(); + } + + @Bean + @ConditionalOnMissingBean + public DSSFrameworkOrchestratorService createDSSFrameworkOrchestratorService() { + return new DSSFrameworkOrchestratorServiceImpl(); + } + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ValidationExceptionMapper.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ValidationExceptionMapper.java new file mode 100644 index 000000000..4b16c6403 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/conf/ValidationExceptionMapper.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.conf; + + + +import com.webank.wedatasphere.linkis.server.Message; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.validation.ConstraintViolation; +import javax.validation.ConstraintViolationException; +import javax.validation.ValidationException; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import javax.ws.rs.ext.ExceptionMapper; +import javax.ws.rs.ext.Provider; + + + +@Provider +public class ValidationExceptionMapper implements ExceptionMapper { + + private final Logger LOGGER = LoggerFactory.getLogger(getClass()); + + @Override + public Response toResponse(ValidationException e) { + if (LOGGER.isDebugEnabled()){ + LOGGER.debug("failed to validate request bean", e); + } + StringBuilder strBuilder = new StringBuilder(); + for (ConstraintViolation cv : ((ConstraintViolationException) e).getConstraintViolations()) { + strBuilder.append(cv.getMessage()).append(";"); + } + Message message = Message.error(strBuilder.toString()); + return Message.messageToResponse(message); + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/OrchestratorTypeEnum.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/OrchestratorTypeEnum.java new file mode 100644 index 000000000..5db47b925 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/OrchestratorTypeEnum.java @@ -0,0 +1,88 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.contant; + +import com.webank.wedatasphere.dss.orchestrator.core.type.OrchestratorKindEnum; +import org.apache.commons.lang.StringUtils; + +import java.util.Arrays; + +public enum OrchestratorTypeEnum { + + WORKFLOW("'pom_work_flow'", 1, "工作流"), + SINGLE_TASK("'pom_single_task'",2, "单任务"), + COMBINED("'pom_consist_orchestrator'",3, "组合编排"); + + private String key; + private Integer type; + private String name; + + OrchestratorTypeEnum(String key, Integer type, String name) { + this.key = key; + this.type = type; + this.name = name; + } + + public String getKey() { + return key; + } + + public void setKey(String key) { + this.key = key; + } + + public Integer getType() { + return type; + } + + public void setType(Integer type) { + this.type = type; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public static Integer getTypeByKey(String key){ + return Arrays.stream(OrchestratorTypeEnum.values()) + .filter(a -> a.getKey().equals(key)) + .map(OrchestratorTypeEnum::getType) + .findFirst() + .orElse(0); + } + + public static String getKeyByOrcType(String type) { + int index = Arrays.stream(OrchestratorKindEnum.values()) + .filter(a -> a.getName().equals(type)) + .map(OrchestratorKindEnum::getIndex) + .findFirst() + .orElse(0); + String key = Arrays.stream(OrchestratorTypeEnum.values()) + .filter(a -> a.getType().equals(index)) + .map(OrchestratorTypeEnum::getKey) + .findFirst() + .orElse(null); + return StringUtils.isNotBlank(key) ? key.replace("'", "") : null; + } + + + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectServerResponse.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectServerResponse.java new file mode 100644 index 000000000..93fbef2e4 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectServerResponse.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.contant; + +public enum ProjectServerResponse { + PROJECT_NOT_EDIT_AUTH(6001,"没有修改权限"), + PROJECT_NOT_EDIT_NAME(6002,"不能修改工程名称"), + PROJECT_NOT_EXIST(6003,"工程不存在"), + PROJECT_IS_NOT_ADMIN(6004,"只有创建人或管理员才能编辑工程") + ; + + ProjectServerResponse(int code, String msg) { + this.code = code; + this.msg = msg; + } + + private int code; + private String msg; + + public int getCode() { + return code; + } + + public void setCode(int code) { + this.code = code; + } + + public String getMsg() { + return msg; + } + + public void setMsg(String msg) { + this.msg = msg; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectSpringConfiguration.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectSpringConfiguration.java new file mode 100644 index 000000000..12f52b7c9 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectSpringConfiguration.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.contant; + +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService; +import com.webank.wedatasphere.dss.framework.project.service.impl.DSSProjectServiceImpl; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class ProjectSpringConfiguration { + + @Bean + @ConditionalOnMissingBean + public DSSProjectService createProjectService() { + return new DSSProjectServiceImpl(); + } + +} \ No newline at end of file diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectUserPrivEnum.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectUserPrivEnum.java new file mode 100644 index 000000000..3de13c31c --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/contant/ProjectUserPrivEnum.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.contant; + + +public enum ProjectUserPrivEnum { + + PRIV_ACCESS(1,"查看权限"), + PRIV_EDIT(2,"编辑权限"), + PRIV_RELEASE(3,"发布权限"); + + ProjectUserPrivEnum(int rank, String name) { + this.rank = rank; + this.name = name; + } + + private int rank; + + private String name; + + public int getRank() { + return rank; + } + + public void setRank(int rank) { + this.rank = rank; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSOrchestratorMapper.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSOrchestratorMapper.java new file mode 100644 index 000000000..b310318b2 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSOrchestratorMapper.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSOrchestrator; +import org.apache.ibatis.annotations.*; + + +@Mapper +public interface DSSOrchestratorMapper extends BaseMapper { + + @Select("Select `uuid` from `dss_orchestrator_info` where `id` = #{orchestratorId} ") + String getUUID(@Param("orchestratorId") Long orchestratorId); +} + + diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSProjectMapper.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSProjectMapper.java new file mode 100644 index 000000000..5cc71f169 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSProjectMapper.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.po.DSSProjectPo; +import com.webank.wedatasphere.dss.framework.project.entity.po.ProjectRelationPo; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectQueryRequest; +import com.webank.wedatasphere.dss.framework.project.entity.vo.ProjectInfoVo; +import com.webank.wedatasphere.dss.framework.project.entity.vo.QueryProjectVo; +import org.apache.ibatis.annotations.Mapper; +import org.apache.ibatis.annotations.Param; +import org.apache.ibatis.annotations.Select; +import org.apache.ibatis.annotations.Update; + +import java.util.List; + + +@Mapper +public interface DSSProjectMapper extends BaseMapper { + + void addProject(DSSProjectPo dssProjectPo); + + @Select("select id from dss_project where `name` = #{projectName}") + Long getProjectIdByName(@Param("projectName") String projectName); + + @Select("select `name` from dss_project where `id` = #{projectId}") + String getProjectNameById(@Param("projectId") Long projectId); + + + @Select("select `id` from dss_project where `workspace_id` = #{workspaceId} and visible = #{visible}") + List getProjectIdsByWorkspaceId(@Param("workspaceId") Long workspaceId,@Param("visible")int visible); + + + List getListByParam(ProjectQueryRequest projectRequest); + + /** + * 获取工程详情：工程名称、空间名称 + */ + ProjectInfoVo getProjectInfoById(@Param("id") Long id); + + void saveProjectRelation(List projectRelationPoList); + + @Select("select `appconn_instance_project_id` from dss_appconn_project_relation " + + "where `project_id` = #{dssProjectId} and `appconn_instance_id` = #{appInstanceId}") + Long getAppConnProjectId(@Param("appInstanceId")Long appInstanceId, @Param("dssProjectId")Long dssProjectId); + + + @Update("update dss_project set `visible` = 0 where `id` = #{projectId}") + void deleteProject(@Param("projectId")Long projectId); + + @Select("SELECT a.url FROM dss_appconn_instance a LEFT JOIN dss_appconn b ON a.appconn_id = b.id WHERE b.appconn_name=#{schedulisName} LIMIT 1") + String getSchedualisUrl(@Param("schedulisName")String schedulisName); + + @Select("SELECT a.appconn_instance_project_id FROM dss_appconn_project_relation a WHERE a.appconn_instance_id = 1 AND a.project_id = #{projectId} LIMIT 1") + Long getSchedulisProjectId(@Param("projectId")Long projectId); + + @Update("update dss_appconn_project_relation set appconn_instance_project_id = #{schedulisProjectId} WHERE appconn_instance_id = 1 AND project_id = #{projectId} ") + int updateSchedulisProjectId(@Param("schedulisProjectId")Long schedulisProjectId,@Param("projectId")Long projectId); +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSProjectUserMapper.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSProjectUserMapper.java new file mode 100644 index 000000000..999ec38ac --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/DSSProjectUserMapper.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectUser; +import org.apache.ibatis.annotations.*; + +import java.util.List; + + +@Mapper +public interface DSSProjectUserMapper extends BaseMapper { + @Insert({ + "" + }) + void insertBatchProjectUser(@Param("list") List projectUserList); + + @Delete("delete from dss_project_user where project_id = #{projectID}") + void deleteAllPriv(@Param("projectID") long projectID); + + @Select("SELECT COUNT(0) FROM dss_workspace_user_role WHERE workspace_id = #{workspaceId} AND username = #{username} AND role_id = #{roleId} ") + Long isAdminByUsername(@Param("workspaceId")Long workspaceId,@Param("username")String username,@Param("roleId")int roleId); +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/impl/DSSProjectMapper.xml b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/impl/DSSProjectMapper.xml new file mode 100644 index 000000000..471308393 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/dao/impl/DSSProjectMapper.xml @@ -0,0 +1,77 @@ + + + + + + + id,`name`,`source`, `workspace_id`,`description`,`org_id`,`visibility`,`is_transfer`,`initial_org_id`, + `username`,`create_time`,`create_by`,`product`,`application_area`,`business` + , `user_id`, `create_by_str`, `update_by_str` + + + + INSERT INTO dss_project () + VALUES + (#{id},#{name},#{source}, #{workspaceId}, #{description},#{orgID},#{visibility},#{isTransfer},#{initialOrgID}, + #{username},#{createTime},#{createBy},#{product},#{applicationArea},#{business}, + #{userID},#{createByStr},#{updateByStr}) + + + + + + + + insert into dss_appconn_project_relation + ( + `project_id`, + `appconn_instance_id`, + `appconn_instance_project_id` + ) + values + + ( + #{item.dssProjectId}, + #{item.appInstanceId}, + #{item.appInstanceProjectId} + ) + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSOrchestrator.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSOrchestrator.java new file mode 100644 index 000000000..6269691e3 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSOrchestrator.java @@ -0,0 +1,236 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableField; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.*; + + +@TableName(value = "dss_project_orchestrator") +public class DSSOrchestrator implements Serializable { + + private static final long serialVersionUID=1L; + + /** + * 主键ID + */ + @TableId(value = "id", type = IdType.AUTO) + private Long id; + + /** + * 空间id + */ + private Long workspaceId; + + /** + * 工程id + */ + private Long projectId; + + /** + * 编排模式id（工作流,调用orchestrator服务返回的orchestratorId） + */ + private Long orchestratorId; + + /** + * 编排模式版本id（工作流,调用orchestrator服务返回的orchestratorVersionId） + */ + private Long orchestratorVersionId; + + /** + * 编排名称 + */ + private String orchestratorName; + + /** + * 编排模式，取得的值是dss_dictionary中的dic_key(parent_key=p_orchestratorment_mode) + */ + private String orchestratorMode; + + /** + * 编排方式 + */ + private String orchestratorWay; + + /** + * 用途 + */ + private String uses; + + /** + * 描述 + */ + private String description; + + /** + * 创建人 + */ + private String createUser; + + /** + * 创建时间 + */ + private Date createTime; + + /** + * 更新人 + */ + private String updateUser; + + /** + * 更新时间 + */ + private Date updateTime; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public Long getOrchestratorVersionId() { + return orchestratorVersionId; + } + + public void setOrchestratorVersionId(Long orchestratorVersionId) { + this.orchestratorVersionId = orchestratorVersionId; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + public String getOrchestratorWay() { + return orchestratorWay; + } + + public void setOrchestratorWay(String orchestratorWay) { + this.orchestratorWay = orchestratorWay; + } + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getCreateUser() { + return createUser; + } + + public void setCreateUser(String createUser) { + this.createUser = createUser; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getUpdateUser() { + return updateUser; + } + + public void setUpdateUser(String updateUser) { + this.updateUser = updateUser; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + @Override + public String toString() { + return "DSSOrchestrator{" + + "id=" + id + + ", workspaceId=" + workspaceId + + ", projectId=" + projectId + + ", orchestratorId=" + orchestratorId + + ", orchestratorVersionId=" + orchestratorVersionId + + ", orchestratorName='" + orchestratorName + '\'' + + ", orchestratorMode='" + orchestratorMode + '\'' + + ", orchestratorWay='" + orchestratorWay + '\'' + + ", uses='" + uses + '\'' + + ", description='" + description + '\'' + + ", createUser='" + createUser + '\'' + + ", createTime=" + createTime + + ", updateUser='" + updateUser + '\'' + + ", updateTime=" + updateTime + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSProjectDO.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSProjectDO.java new file mode 100644 index 000000000..a26f72a5e --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSProjectDO.java @@ -0,0 +1,344 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableField; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + + +@TableName(value = "dss_project") +public class DSSProjectDO implements Serializable { + + private static final long serialVersionUID=1L; + + @TableId(value = "id", type = IdType.AUTO) + private Long id; + + private String name; + + /** + * Source of the dss_project + */ + private String source; + + private String description; + + private Long userId; + + private String username; + + private Long workspaceId; + + private Date createTime; + + private String createBy; + + private Date updateTime; + + private String updateBy; + + /** + * Organization ID + */ + private Long orgId; + + private Boolean visibility; + + /** + * Reserved word + */ + private Boolean isTransfer; + + private Long initialOrgId; + + /** + * If it is archived + */ + @TableField("isArchive") + private Boolean isArchive; + + private String pic; + + private Integer starNum; + + private String product; + + private Integer applicationArea; + + private String business; + + private Integer isPersonal; + + private String createByStr; + + private String updateByStr; + /** + * 开发流程，多个以英文逗号分隔，取得的值是dss_dictionary中的dic_key(parent_key=p_develop_process)，首尾以英文逗号结束 + */ + private String devProcess; + + /** + * 编码模式，多个以英文逗号分隔，取得的值是dss_dictionary中的dic_key(parent_key=p_arrangement_mode或下面一级)，首尾以英文逗号结束 + */ + private String orchestratorMode; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Long getUserId() { + return userId; + } + + public void setUserId(Long userId) { + this.userId = userId; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getCreateBy() { + return createBy; + } + + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdateBy() { + return updateBy; + } + + public void setUpdateBy(String updateBy) { + this.updateBy = updateBy; + } + + public Long getOrgId() { + return orgId; + } + + public void setOrgId(Long orgId) { + this.orgId = orgId; + } + + public Boolean getVisibility() { + return visibility; + } + + public void setVisibility(Boolean visibility) { + this.visibility = visibility; + } + + public Boolean getTransfer() { + return isTransfer; + } + + public void setTransfer(Boolean transfer) { + isTransfer = transfer; + } + + public Long getInitialOrgId() { + return initialOrgId; + } + + public void setInitialOrgId(Long initialOrgId) { + this.initialOrgId = initialOrgId; + } + + public Boolean getArchive() { + return isArchive; + } + + public void setArchive(Boolean archive) { + isArchive = archive; + } + + public String getPic() { + return pic; + } + + public void setPic(String pic) { + this.pic = pic; + } + + public Integer getStarNum() { + return starNum; + } + + public void setStarNum(Integer starNum) { + this.starNum = starNum; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public Integer getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(Integer applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public Integer getIsPersonal() { + return isPersonal; + } + + public void setIsPersonal(Integer isPersonal) { + this.isPersonal = isPersonal; + } + + public String getCreateByStr() { + return createByStr; + } + + public void setCreateByStr(String createByStr) { + this.createByStr = createByStr; + } + + public String getUpdateByStr() { + return updateByStr; + } + + public void setUpdateByStr(String updateByStr) { + this.updateByStr = updateByStr; + } + + public String getDevProcess() { + return devProcess; + } + + public void setDevProcess(String devProcess) { + this.devProcess = devProcess; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + @Override + public String toString() { + return "DSSProject{" + + "id=" + id + + ", name='" + name + '\'' + + ", source='" + source + '\'' + + ", description='" + description + '\'' + + ", userId=" + userId + + ", username='" + username + '\'' + + ", workspaceId=" + workspaceId + + ", createTime=" + createTime + + ", createBy='" + createBy + '\'' + + ", updateTime=" + updateTime + + ", updateBy='" + updateBy + '\'' + + ", orgId=" + orgId + + ", visibility=" + visibility + + ", isTransfer=" + isTransfer + + ", initialOrgId=" + initialOrgId + + ", isArchive=" + isArchive + + ", pic='" + pic + '\'' + + ", starNum=" + starNum + + ", product='" + product + '\'' + + ", applicationArea=" + applicationArea + + ", business='" + business + '\'' + + ", isPersonal=" + isPersonal + + ", createByStr='" + createByStr + '\'' + + ", updateByStr='" + updateByStr + '\'' + + ", devProcess='" + devProcess + '\'' + + ", orchestratorMode='" + orchestratorMode + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSProjectUser.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSProjectUser.java new file mode 100644 index 000000000..9494d501e --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/DSSProjectUser.java @@ -0,0 +1,106 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + +@TableName(value = "dss_project_user") +public class DSSProjectUser implements Serializable { + + private static final long serialVersionUID=1L; + + @TableId(value = "id", type = IdType.AUTO) + private Long id; + + private Long projectId; + + private String username; + + private Long workspaceId; + + /** + * 权限等级：1-查看，2-编辑，3-发布 + */ + private Integer priv; + + private Date lastUpdateTime; + + public DSSProjectUser() { + } + + + public DSSProjectUser(Long workspaceId, Long projectId, String username, int priv) { + this.workspaceId = workspaceId; + this.projectId = projectId; + this.username = username; + this.priv = priv; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Integer getPriv() { + return priv; + } + + public void setPriv(Integer priv) { + this.priv = priv; + } + + public Date getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(Date lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/po/DSSProjectPo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/po/DSSProjectPo.java new file mode 100644 index 000000000..f6d6513ec --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/po/DSSProjectPo.java @@ -0,0 +1,276 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.po; + +import java.util.Date; +import java.util.List; + + +public class DSSProjectPo { + + + private Long id; + private String name; + private String description; + private String source; + private Date createTime; + private String createBy; //兼容Visualis 后修改类型 + private Date updateTime; + private Long orgID; + private Boolean visibility; + private Boolean isTransfer; + private Boolean isArchive; + private Long initialOrgID; + private String pic; + private Long starNum; + private String product; + private Integer applicationArea; + private String business; + private Integer workspaceId; + private String projectGroup; + private String workspaceName; + private String username; + private Integer isPersonal; + + private Long userID; + private String createByStr; + private String updateByStr; + + private List accessUsers; + private List editUsers; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getCreateBy() { + return createBy; + } + + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public Long getOrgID() { + return orgID; + } + + public void setOrgID(Long orgID) { + this.orgID = orgID; + } + + public Boolean getVisibility() { + return visibility; + } + + public void setVisibility(Boolean visibility) { + this.visibility = visibility; + } + + public Boolean getTransfer() { + return isTransfer; + } + + public void setTransfer(Boolean transfer) { + isTransfer = transfer; + } + + public Boolean getArchive() { + return isArchive; + } + + public void setArchive(Boolean archive) { + isArchive = archive; + } + + public Long getInitialOrgID() { + return initialOrgID; + } + + public void setInitialOrgID(Long initialOrgID) { + this.initialOrgID = initialOrgID; + } + + public String getPic() { + return pic; + } + + public void setPic(String pic) { + this.pic = pic; + } + + public Long getStarNum() { + return starNum; + } + + public void setStarNum(Long starNum) { + this.starNum = starNum; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public Integer getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(Integer applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public String getProjectGroup() { + return projectGroup; + } + + public void setProjectGroup(String projectGroup) { + this.projectGroup = projectGroup; + } + + public List getAccessUsers() { + return accessUsers; + } + + public void setAccessUsers(List accessUsers) { + this.accessUsers = accessUsers; + } + + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public Integer getIsPersonal() { + return isPersonal; + } + + public void setIsPersonal(Integer isPersonal) { + this.isPersonal = isPersonal; + } + + public Long getUserID() { + return userID; + } + + public void setUserID(Long userID) { + this.userID = userID; + } + + public String getCreateByStr() { + return createByStr; + } + + public void setCreateByStr(String createByStr) { + this.createByStr = createByStr; + } + + public String getUpdateByStr() { + return updateByStr; + } + + public void setUpdateByStr(String updateByStr) { + this.updateByStr = updateByStr; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/po/ProjectRelationPo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/po/ProjectRelationPo.java new file mode 100644 index 000000000..d7648d0f4 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/po/ProjectRelationPo.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.po; + + +public class ProjectRelationPo { + + private Long dssProjectId; + + private Long appInstanceId; + + private Long appInstanceProjectId; + + + public ProjectRelationPo(Long dssProjectId, Long appInstanceId, Long appInstanceProjectId) { + this.dssProjectId = dssProjectId; + this.appInstanceId = appInstanceId; + this.appInstanceProjectId = appInstanceProjectId; + } + + public Long getDssProjectId() { + return dssProjectId; + } + + public void setDssProjectId(Long dssProjectId) { + this.dssProjectId = dssProjectId; + } + + public Long getAppInstanceId() { + return appInstanceId; + } + + public void setAppInstanceId(Long appInstanceId) { + this.appInstanceId = appInstanceId; + } + + public Long getAppInstanceProjectId() { + return appInstanceProjectId; + } + + public void setAppInstanceProjectId(Long appInstanceProjectId) { + this.appInstanceProjectId = appInstanceProjectId; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ForbiddenFlowRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ForbiddenFlowRequest.java new file mode 100644 index 000000000..91e7eff88 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ForbiddenFlowRequest.java @@ -0,0 +1,79 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; + + +@XmlRootElement +public class ForbiddenFlowRequest { + + @NotNull(message = "编排模式Id不能为空") + private Long orchestratorId; + + @NotNull(message = "工程id不能为空") + private Long projectId; + + @NotNull(message = "禁用标示不能为空") + private String activeFlag; + + @NotNull(message = "工作流名称不能为空") + private String orchestratorName; + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public String getActiveFlag() { + return activeFlag; + } + + public void setActiveFlag(String activeFlag) { + this.activeFlag = activeFlag; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + @Override + public String toString() { + return "ForbiddenFlowRequest{" + + "orchestratorId=" + orchestratorId + + ", projectId=" + projectId + + ", activeFlag='" + activeFlag + '\'' + + ", orchestratorName='" + orchestratorName + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorCreateRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorCreateRequest.java new file mode 100644 index 000000000..9fbd697d4 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorCreateRequest.java @@ -0,0 +1,161 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import com.webank.wedatasphere.dss.framework.project.entity.vo.LabelRouteVo; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class OrchestratorCreateRequest { + + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + @NotNull(message = "工程id不能为空") + private Long projectId; + + @NotNull(message = "编排名称不能为空") + private String orchestratorName; + + /** + * 编排模式，如工作流，组合编排等 + */ + @NotNull(message = "编排模式类型不能为空") + private String orchestratorMode; + + /** + * 编排方式 + */ + @NotNull(message = "编排方式不能为空") + private List orchestratorWays; + + /** + * 编排用途 + */ + private String uses; + + @NotNull(message = "描述不能为空") + private String description; + + + private String projectName; + + private String workspaceName; + + /** + * labels是通过前端进行传入的，主要是用来进行当前的环境信息 + */ + private LabelRouteVo labels; + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + public List getOrchestratorWays() { + return orchestratorWays; + } + + public void setOrchestratorWays(List orchestratorWays) { + this.orchestratorWays = orchestratorWays; + } + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public LabelRouteVo getLabels() { + return labels; + } + + public void setLabels(LabelRouteVo labels) { + this.labels = labels; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + @Override + public String toString() { + return "OrchestratorCreateRequest{" + + "workspaceId=" + workspaceId + + ", projectId=" + projectId + + ", arrangeName='" + orchestratorName + '\'' + + ", arrangeMode='" + orchestratorMode + '\'' + + ", arrangeWays=" + orchestratorWays + + ", uses='" + uses + '\'' + + ", description='" + description + '\'' + + ", labels=" + labels + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorDeleteRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorDeleteRequest.java new file mode 100644 index 000000000..15f79847c --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorDeleteRequest.java @@ -0,0 +1,74 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import com.webank.wedatasphere.dss.framework.project.entity.vo.LabelRouteVo; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class OrchestratorDeleteRequest { + + @NotNull(message = "id不能为空") + private Long id; + + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + @NotNull(message = "工程id不能为空") + private Long projectId; + + /** + * dssLabels是通过前端进行传入的，主要是用来进行当前的环境信息 + */ + private LabelRouteVo labels; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public LabelRouteVo getLabels() { + return labels; + } + + public void setLabels(LabelRouteVo labels) { + this.labels = labels; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorModifyRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorModifyRequest.java new file mode 100644 index 000000000..2a8a9c9e3 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorModifyRequest.java @@ -0,0 +1,152 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import com.webank.wedatasphere.dss.framework.project.entity.vo.LabelRouteVo; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class OrchestratorModifyRequest { + + @NotNull(message = "id不能为空") + private Long id; + + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + @NotNull(message = "工程id不能为空") + private Long projectId; + + @NotNull(message = "编排名称不能为空") + private String orchestratorName; + + /** + * 编排模式，如工作流，组合编排等 + */ + @NotNull(message = "编排模式不能为空") + private String orchestratorMode; + + /** + * 编排方式 + */ + @NotNull(message = "编排方式不能为空") + private List orchestratorWays; + + /** + * 编排用途 + */ + private String uses; + + @NotNull(message = "描述不能为空") + private String description; + + /** + * labels是通过前端进行传入的，主要是用来进行当前的环境信息 + */ + private LabelRouteVo labels; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + public List getOrchestratorWays() { + return orchestratorWays; + } + + public void setOrchestratorWays(List orchestratorWays) { + this.orchestratorWays = orchestratorWays; + } + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public LabelRouteVo getLabels() { + return labels; + } + + public void setLabels(LabelRouteVo labels) { + this.labels = labels; + } + + @Override + public String toString() { + return "OrchestratorModifyRequest{" + + "id=" + id + + ", workspaceId=" + workspaceId + + ", projectId=" + projectId + + ", orchestratorName='" + orchestratorName + '\'' + + ", orchestratorMode='" + orchestratorMode + '\'' + + ", orchestratorWays=" + orchestratorWays + + ", uses='" + uses + '\'' + + ", description='" + description + '\'' + + ", labels=" + labels + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorRequest.java new file mode 100644 index 000000000..235e5485d --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/OrchestratorRequest.java @@ -0,0 +1,83 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class OrchestratorRequest { + + private Long id; + + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + @NotNull(message = "工程id不能为空") + private Long projectId; + + /** + * 编排类型，如工作流，组合编排等 + */ + // @NotNull(message = "编排类型不能为空") + private String orchestratorMode; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + @Override + public String toString() { + return "OrchestratorRequest{" + + "id=" + id + + ", workspaceId=" + workspaceId + + ", projectId=" + projectId + + ", orchestratorMode='" + orchestratorMode + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectCreateRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectCreateRequest.java new file mode 100644 index 000000000..4d9710881 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectCreateRequest.java @@ -0,0 +1,160 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class ProjectCreateRequest { + + @NotNull(message = "工程名称不能为空") + private String name; + + @NotNull(message = "应用领域不能为空") + private Integer applicationArea; + + private String business; + + //产品 + private String product; + + private List releaseUsers; + + private List editUsers; + + private List accessUsers; + + @NotNull(message = "工程描述不能为空") + private String description; + + /** + * 工作空间名,因为是全局唯一的 + */ + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + private String workspaceName; + /** + * 开发流程 list + */ + private List devProcessList; + + /** + * 编码模式 list + */ + private List orchestratorModeList; + + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Integer getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(Integer applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public List getReleaseUsers() { + return releaseUsers; + } + + public void setReleaseUsers(List releaseUsers) { + this.releaseUsers = releaseUsers; + } + + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + public List getAccessUsers() { + return accessUsers; + } + + public void setAccessUsers(List accessUsers) { + this.accessUsers = accessUsers; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public List getDevProcessList() { + return devProcessList; + } + + public void setDevProcessList(List devProcessList) { + this.devProcessList = devProcessList; + } + + public List getOrchestratorModeList() { + return orchestratorModeList; + } + + public void setOrchestratorModeList(List orchestratorModeList) { + this.orchestratorModeList = orchestratorModeList; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectDeleteRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectDeleteRequest.java new file mode 100644 index 000000000..031521267 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectDeleteRequest.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; + + +@XmlRootElement +public class ProjectDeleteRequest { + + @NotNull(message = "工程id不能为空") + private Long id; + + @NotNull(message = "确认不能为空") + private boolean sure; + + @NotNull(message = "同步删除其它系统工程不能为空") + private boolean ifDelOtherSys; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public boolean isSure() { + return sure; + } + + public void setSure(boolean sure) { + this.sure = sure; + } + + public boolean isIfDelOtherSys() { + return ifDelOtherSys; + } + + public void setIfDelOtherSys(boolean ifDelOtherSys) { + this.ifDelOtherSys = ifDelOtherSys; + } + + @Override + public String toString() { + return "ProjectDeleteRequest{" + + "id=" + id + + ", sure=" + sure + + ", ifDelOtherSys=" + ifDelOtherSys + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectModifyRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectModifyRequest.java new file mode 100644 index 000000000..655e63c38 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectModifyRequest.java @@ -0,0 +1,184 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class ProjectModifyRequest { + + + @NotNull(message = "工程名称不能为空") + private String name; + + @NotNull(message = "工程id不能为空") + private Long id; + + @NotNull(message = "应用领域不能为空") + private String applicationArea; + + //业务 + private String business; + + //编辑权限用户 + private List editUsers; + + //查看权限用户 + private List accessUsers; + + //发布权限用户 + private List releaseUsers; + + @NotNull(message = "工程描述不能为空") + private String description; + + //产品 + private String product; + + /** + * 工作空间名,因为是全局唯一的 + */ + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + /* + 开发流程 code list + */ + private List devProcessList; + + /** + * 编排模式 code list + */ + private List orchestratorModeList; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(String applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + public List getAccessUsers() { + return accessUsers; + } + + public void setAccessUsers(List accessUsers) { + this.accessUsers = accessUsers; + } + + public List getReleaseUsers() { + return releaseUsers; + } + + public void setReleaseUsers(List releaseUsers) { + this.releaseUsers = releaseUsers; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public List getDevProcessList() { + return devProcessList; + } + + public void setDevProcessList(List devProcessList) { + this.devProcessList = devProcessList; + } + + public List getOrchestratorModeList() { + return orchestratorModeList; + } + + public void setOrchestratorModeList(List orchestratorModeList) { + this.orchestratorModeList = orchestratorModeList; + } + + @Override + public String toString() { + return "ProjectModifyRequest{" + + "name='" + name + '\'' + + ", id=" + id + + ", applicationArea='" + applicationArea + '\'' + + ", business='" + business + '\'' + + ", editUsers=" + editUsers + + ", accessUsers=" + accessUsers + + ", releaseUsers=" + releaseUsers + + ", description='" + description + '\'' + + ", product='" + product + '\'' + + ", workspaceId=" + workspaceId + + ", devProcessList=" + devProcessList + + ", orchestratorModeList=" + orchestratorModeList + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectQueryRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectQueryRequest.java new file mode 100644 index 000000000..b0147415a --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/ProjectQueryRequest.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.io.Serializable; + + +@XmlRootElement +public class ProjectQueryRequest implements Serializable { + + private static final long serialVersionUID=1L; + + private Long id; + + @NotNull(message = "workspaceId不能为空") + private Long workspaceId; + + private String username; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/RemoveFlowRequest.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/RemoveFlowRequest.java new file mode 100644 index 000000000..d1f7cfd83 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/request/RemoveFlowRequest.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.request; + +import javax.validation.constraints.NotNull; +import javax.xml.bind.annotation.XmlRootElement; +import java.util.List; + + +@XmlRootElement +public class RemoveFlowRequest { + + private Long orchestratorId; + + @NotNull(message = "工程id不能为空") + private Long projectId; + + @NotNull(message = "工作流名称不能为空") + private String orchestratorName; + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + @Override + public String toString() { + return "RemoveFlowRequest{" + + "orchestratorId=" + orchestratorId + + ", projectId=" + projectId + + ", orchestratorName='" + orchestratorName + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/response/ProjectResponse.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/response/ProjectResponse.java new file mode 100644 index 000000000..9344ee01c --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/response/ProjectResponse.java @@ -0,0 +1,222 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.response; + +import java.io.Serializable; +import java.util.Date; +import java.util.List; + + +public class ProjectResponse implements Serializable { + private static final long serialVersionUID=1L; + //工程id + private Long id; + //应用领域 + private Integer applicationArea; + //业务 + private String business; + //创建人 + private String createBy; + //工程描述 + private String description; + //工程名称 + private String name; + // + private String source; + //产品 + private String product; + private Boolean isArchive; + //工程创建时间 + private Date createTime; + //工程修改时间 + private Date updateTime; + + /** + * 发布用户 list + */ + private List releaseUsers; + + /** + * 编辑用户 list + */ + private List editUsers; + + /** + * 查看用户 list + */ + private List accessUsers; + + /** + * 开发流程 list + */ + private List devProcessList; + + /** + * 编码模式 list + */ + private List orchestratorModeList; + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Integer getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(Integer applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public String getCreateBy() { + return createBy; + } + + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public Boolean getArchive() { + return isArchive; + } + + public void setArchive(Boolean archive) { + isArchive = archive; + } + + public List getReleaseUsers() { + return releaseUsers; + } + + public void setReleaseUsers(List releaseUsers) { + this.releaseUsers = releaseUsers; + } + + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + public List getAccessUsers() { + return accessUsers; + } + + public void setAccessUsers(List accessUsers) { + this.accessUsers = accessUsers; + } + + public List getDevProcessList() { + return devProcessList; + } + + public void setDevProcessList(List devProcessList) { + this.devProcessList = devProcessList; + } + + public List getOrchestratorModeList() { + return orchestratorModeList; + } + + public void setOrchestratorModeList(List orchestratorModeList) { + this.orchestratorModeList = orchestratorModeList; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + @Override + public String toString() { + return "ResponseProjectVo{" + + "id=" + id + + ", applicationArea=" + applicationArea + + ", business='" + business + '\'' + + ", createBy='" + createBy + '\'' + + ", description='" + description + '\'' + + ", name='" + name + '\'' + + ", source='" + source + '\'' + + ", product='" + product + '\'' + + ", isArchive=" + isArchive + + ", createTime=" + createTime + + ", updateTime=" + updateTime + + ", releaseUsers=" + releaseUsers + + ", editUsers=" + editUsers + + ", accessUsers=" + accessUsers + + ", devProcessList=" + devProcessList + + ", orchestratorModeList=" + orchestratorModeList + + '}'; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/AbstractOrchestratorVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/AbstractOrchestratorVo.java new file mode 100644 index 000000000..5a32ef18f --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/AbstractOrchestratorVo.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + + +public abstract class AbstractOrchestratorVo { + + final static String WORK_FLOW = "workflow"; + + final static String ORDINARY = "ordinary"; + + private Long orchestratorId; + + private String orchestratorVersion; + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public String getOrchestratorVersion() { + return orchestratorVersion; + } + + public void setOrchestratorVersion(String orchestratorVersion) { + this.orchestratorVersion = orchestratorVersion; + } + + abstract String getType(); + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/CommonOrchestratorVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/CommonOrchestratorVo.java new file mode 100644 index 000000000..93700f75a --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/CommonOrchestratorVo.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + + +public class CommonOrchestratorVo extends AbstractOrchestratorVo{ + + @Override + String getType() { + return WORK_FLOW; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSFlowOrchestratorFlowVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSFlowOrchestratorFlowVo.java new file mode 100644 index 000000000..fa231af96 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSFlowOrchestratorFlowVo.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +import java.util.List; + + +public class DSSFlowOrchestratorFlowVo { + + private Integer orchestratorId; + + private String orchestratorVersion; + + /** + * flowId 传递到前端让前端使用进行工作流的渲染 + */ + private String flowId; + + private String description; + + private String name; + + private List tags; + + public Integer getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Integer orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public String getOrchestratorVersion() { + return orchestratorVersion; + } + + public void setOrchestratorVersion(String orchestratorVersion) { + this.orchestratorVersion = orchestratorVersion; + } + + public String getFlowId() { + return flowId; + } + + public void setFlowId(String flowId) { + this.flowId = flowId; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public List getTags() { + return tags; + } + + public void setTags(List tags) { + this.tags = tags; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSProjectDetailVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSProjectDetailVo.java new file mode 100644 index 000000000..cf5a85033 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSProjectDetailVo.java @@ -0,0 +1,109 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +import java.util.List; + + + +public class DSSProjectDetailVo extends DSSProjectVo { + + /** + * 发布用户 list + */ + private List releaseUsers; + + /** + * 编辑用户 list + */ + private List editUsers; + + /** + * 查看用户 list + */ + private List accessUsers; + + /** + * 开发的模式，主要是工作流和应用工具 + */ + private String projectMode; + + /** + * 应用领域，不为空 + */ + private String applicationArea; + + + /** + * 使用业务 + */ + private String business; + + + private String description; + + + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + public List getAccessUsers() { + return accessUsers; + } + + public void setAccessUsers(List accessUsers) { + this.accessUsers = accessUsers; + } + + public String getProjectMode() { + return projectMode; + } + + public void setProjectMode(String projectMode) { + this.projectMode = projectMode; + } + + public String getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(String applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + @Override + public String getDescription() { + return description; + } + + @Override + public void setDescription(String description) { + this.description = description; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSProjectVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSProjectVo.java new file mode 100644 index 000000000..a7206d452 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/DSSProjectVo.java @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +import java.util.List; + + +public class DSSProjectVo { + + private Long id; + + private String name; + + private String description; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/LabelRouteVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/LabelRouteVo.java new file mode 100644 index 000000000..621a99b2b --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/LabelRouteVo.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +public class LabelRouteVo { + + private String route; + + public String getRoute() { + return route; + } + + public void setRoute(String route) { + this.route = route; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/OrchestratorBaseInfo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/OrchestratorBaseInfo.java new file mode 100644 index 000000000..63a3e45f2 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/OrchestratorBaseInfo.java @@ -0,0 +1,229 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +import java.util.Date; +import java.util.List; + +public class OrchestratorBaseInfo { + /** + * 主键ID + */ + private Long id; + + /** + * 空间id + */ + private Long workspaceId; + + /** + * 工程id + */ + private Long projectId; + + /** + * 编排模式id（工作流,调用orchestrator服务返回的orchestratorId） + */ + private Long orchestratorId; + + /** + * 编排模式版本id（工作流,调用orchestrator服务返回的orchestratorVersionId） + */ + private Long orchestratorVersionId; + + /** + * 编排名称 + */ + private String orchestratorName; + + /** + * 编排模式，取得的值是dss_dictionary中的dic_key(parent_key=p_orchestratorment_mode) + */ + private String orchestratorMode; + + /** + * 编排方式 + */ + //private String orchestratorWay; + + /** + * 用途 + */ + private String uses; + + /** + * 描述 + */ + private String description; + + /** + * 创建人 + */ + private String createUser; + + /** + * 创建时间 + */ + private Date createTime; + + /** + * 更新人 + */ + private String updateUser; + + /** + * 更新时间 + */ + private Date updateTime; + + private List orchestratorWays; + + private boolean flowEditLockExist=false; + + /** + * 工程权限等级：0-查看，1-编辑，2-发布 + */ + private Integer priv; + + public Integer getPriv() { + return priv; + } + + public void setPriv(Integer priv) { + this.priv = priv; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public Long getOrchestratorVersionId() { + return orchestratorVersionId; + } + + public void setOrchestratorVersionId(Long orchestratorVersionId) { + this.orchestratorVersionId = orchestratorVersionId; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getCreateUser() { + return createUser; + } + + public void setCreateUser(String createUser) { + this.createUser = createUser; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getUpdateUser() { + return updateUser; + } + + public void setUpdateUser(String updateUser) { + this.updateUser = updateUser; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public List getOrchestratorWays() { + return orchestratorWays; + } + + public void setOrchestratorWays(List orchestratorWays) { + this.orchestratorWays = orchestratorWays; + } + + public boolean isFlowEditLockExist() { + return flowEditLockExist; + } + + public void setFlowEditLockExist(boolean flowEditLockExist) { + this.flowEditLockExist = flowEditLockExist; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/OrdinaryOrchestratorVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/OrdinaryOrchestratorVo.java new file mode 100644 index 000000000..4c643f8bb --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/OrdinaryOrchestratorVo.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + + +public class OrdinaryOrchestratorVo extends AbstractOrchestratorVo{ + + @Override + String getType() { + return ORDINARY; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/ProcessNode.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/ProcessNode.java new file mode 100644 index 000000000..094c32a85 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/ProcessNode.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + + +public interface ProcessNode { + + Integer getIndex(); + + String getName(); + + String getChName(); + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/ProjectInfoVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/ProjectInfoVo.java new file mode 100644 index 000000000..abe420c48 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/ProjectInfoVo.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +import java.io.Serializable; + + +public class ProjectInfoVo implements Serializable { + + private static final long serialVersionUID=1L; + private Long id; + private Long workspaceId; + private String projectName; + private String workspaceName; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/QueryProjectVo.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/QueryProjectVo.java new file mode 100644 index 000000000..687b6d361 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/entity/vo/QueryProjectVo.java @@ -0,0 +1,165 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.entity.vo; + +import java.io.Serializable; +import java.util.Date; +import java.util.List; + + +public class QueryProjectVo implements Serializable { + + private static final long serialVersionUID=1L; + private Long id; + private Integer applicationArea; + private String business; + private String createBy; + private String description; + private String name; + private String source; + private String product; + private Boolean isArchive; + private Date createTime; + private Date updateTime; + + private String devProcess; + private String orchestratorMode; + private String pusername; + + private Integer visible; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Integer getApplicationArea() { + return applicationArea; + } + + public void setApplicationArea(Integer applicationArea) { + this.applicationArea = applicationArea; + } + + public String getBusiness() { + return business; + } + + public void setBusiness(String business) { + this.business = business; + } + + public String getCreateBy() { + return createBy; + } + + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public Boolean getArchive() { + return isArchive; + } + + public void setArchive(Boolean archive) { + isArchive = archive; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getDevProcess() { + return devProcess; + } + + public void setDevProcess(String devProcess) { + this.devProcess = devProcess; + } + + public String getOrchestratorMode() { + return orchestratorMode; + } + + public void setOrchestratorMode(String orchestratorMode) { + this.orchestratorMode = orchestratorMode; + } + + public String getPusername() { + return pusername; + } + + public void setPusername(String pusername) { + this.pusername = pusername; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public Integer getVisible() { + return visible; + } + + public void setVisible(Integer visible) { + this.visible = visible; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/DSSProjectErrorException.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/DSSProjectErrorException.java new file mode 100644 index 000000000..9e291f489 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/DSSProjectErrorException.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class DSSProjectErrorException extends ErrorException { + + public DSSProjectErrorException(int errorCode, String errorDesc){ + super(errorCode, errorDesc); + } + + + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/DSSProjectWarnException.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/DSSProjectWarnException.java new file mode 100644 index 000000000..175271ab6 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/DSSProjectWarnException.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.exception; + +import com.webank.wedatasphere.linkis.common.exception.WarnException; + + +public class DSSProjectWarnException extends WarnException { + public DSSProjectWarnException(int errorCode, String errorDesc){ + super(errorCode, errorDesc); + } + + + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/LambdaWarnException.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/LambdaWarnException.java new file mode 100644 index 000000000..f14debcd0 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/exception/LambdaWarnException.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.exception; + + +public class LambdaWarnException extends DSSProjectWarnException { + + + public LambdaWarnException(int errorCode, String errorDesc) { + super(errorCode, errorDesc); + } + + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/restful/DSSFrameworkOrchestratorRestful.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/restful/DSSFrameworkOrchestratorRestful.java new file mode 100644 index 000000000..6591dad5e --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/restful/DSSFrameworkOrchestratorRestful.java @@ -0,0 +1,149 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.restful; + +import com.webank.wedatasphere.dss.framework.common.utils.RestfulUtils; +import com.webank.wedatasphere.dss.framework.project.dao.DSSProjectMapper; +import com.webank.wedatasphere.dss.framework.project.entity.request.*; +import com.webank.wedatasphere.dss.framework.project.entity.vo.CommonOrchestratorVo; +import com.webank.wedatasphere.dss.framework.project.service.DSSFrameworkOrchestratorService; +import com.webank.wedatasphere.dss.framework.project.service.DSSOrchestratorService; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.math3.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.validation.Valid; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; + + +@Component +@Path("/dss/framework/project") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSFrameworkOrchestratorRestful { + + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSFrameworkOrchestratorRestful.class); + + @Autowired + private DSSFrameworkOrchestratorService dssFrameworkOrchestratorService; + @Autowired + private DSSOrchestratorService orchestratorService; + + @Autowired + private DSSProjectMapper projectMapper; + + /** + * 创建编排模式 + * + * @param httpServletRequest + * @param createRequest + * @return + */ + @POST + @Path("createOrchestrator") + public Response createOrchestrator(@Context HttpServletRequest httpServletRequest, @Valid OrchestratorCreateRequest createRequest) { + String username = SecurityFilter.getLoginUsername(httpServletRequest); + Workspace workspace = SSOHelper.getWorkspace(httpServletRequest); + LOGGER.info("workspace is {}", workspace.getWorkspaceName()); + try { + //保存编排模式 + //todo 先注释掉 + // orchestratorService.saveOrchestrator(createRequest,null,username); + // return RestfulUtils.dealOk("创建工作流编排模式成功", new Pair<>("orchestratorId", 1L)); + CommonOrchestratorVo orchestratorVo = dssFrameworkOrchestratorService.createOrchestrator(username, createRequest, workspace); + return RestfulUtils.dealOk("创建工作流编排模式成功", new Pair<>("orchestratorId", orchestratorVo.getOrchestratorId())); + } catch (Exception e) { + LOGGER.error("Failed to create orchestrator {} for user {}", createRequest, username, e); + return RestfulUtils.dealError("创建工作流编排模式失败:" + e.getMessage()); + } + } + + /** + * 查询所有的编排模式 + * + * @param httpServletRequest + * @param orchestratorRequest + * @return + */ + @POST + @Path("getAllOrchestrator") + public Response getAllOrchestrator(@Context HttpServletRequest httpServletRequest, @Valid OrchestratorRequest orchestratorRequest) { + try { + String username = SecurityFilter.getLoginUsername(httpServletRequest); + return RestfulUtils.dealOk("获取编排模式成功", new Pair<>("page", orchestratorService.getListByPage(orchestratorRequest, username))); + } catch (Exception e) { + LOGGER.error("getAllOrchestratorError ", e); + return RestfulUtils.dealError("获取编排模式失败:" + e.getMessage()); + } + } + + /** + * 修改编排模式 + * + * @param httpServletRequest + * @param modifyRequest + * @return + */ + @POST + @Path("modifyOrchestrator") + public Response modifyOrchestrator(@Context HttpServletRequest httpServletRequest, @Valid OrchestratorModifyRequest modifyRequest) { + String username = SecurityFilter.getLoginUsername(httpServletRequest); + Workspace workspace = SSOHelper.getWorkspace(httpServletRequest); + LOGGER.info("workspace is {}", workspace.getWorkspaceName()); + try { + CommonOrchestratorVo orchestratorVo = dssFrameworkOrchestratorService.modifyOrchestrator(username, modifyRequest,workspace); + //保存编排模式 +// orchestratorService.updateOrchestrator(modifyRequest, username); + return RestfulUtils.dealOk("创建工作流编排模式成功", new Pair<>("orchestratorId", orchestratorVo.getOrchestratorId())); + } catch (Exception e) { + LOGGER.error("Failed to create orchestrator {} for user {}", modifyRequest, username, e); + return RestfulUtils.dealError("创建工作流编排模式失败:" + e.getMessage()); + } + } + + /** + * 删除编排模式 + * + * @param httpServletRequest + * @param deleteRequest + * @return + */ + @POST + @Path("deleteOrchestrator") + public Response deleteOrchestrator(@Context HttpServletRequest httpServletRequest, @Valid OrchestratorDeleteRequest deleteRequest) { + String username = SecurityFilter.getLoginUsername(httpServletRequest); + Workspace workspace = SSOHelper.getWorkspace(httpServletRequest); + LOGGER.info("workspace is {}", workspace.getWorkspaceName()); + try { + dssFrameworkOrchestratorService.deleteOrchestrator(username, deleteRequest,workspace); + return RestfulUtils.dealOk("删除工作流编排模式成功"); + } catch (Exception e) { + LOGGER.error("Failed to delete orchestrator {} for user {}", deleteRequest, username, e); + return RestfulUtils.dealError("删除工作流编排模式失败:" + e.getMessage()); + } + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/restful/DSSFrameworkProjectRestfulApi.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/restful/DSSFrameworkProjectRestfulApi.java new file mode 100644 index 000000000..258e04455 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/restful/DSSFrameworkProjectRestfulApi.java @@ -0,0 +1,188 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.restful; + +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectDeleteRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectQueryRequest; +import com.webank.wedatasphere.dss.framework.project.entity.response.ProjectResponse; +import com.webank.wedatasphere.dss.framework.project.entity.vo.DSSProjectVo; +import com.webank.wedatasphere.dss.framework.project.service.DSSFrameworkProjectService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService; +import com.webank.wedatasphere.dss.framework.project.utils.ApplicationArea; +import com.webank.wedatasphere.dss.framework.project.utils.RestfulUtils; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import javax.servlet.http.HttpServletRequest; +import javax.validation.Valid; +import javax.ws.rs.Consumes; +import javax.ws.rs.GET; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import org.apache.commons.math3.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +@Component +@Path("/dss/framework/project") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSFrameworkProjectRestfulApi { + private static final Logger LOGGER = LoggerFactory.getLogger(DSSFrameworkProjectRestfulApi.class); + @Autowired + DSSFrameworkProjectService dssFrameworkProjectService; + @Autowired + private DSSProjectService projectService; + + /** + * 获取所有工程或者单个工程 + * + * @param request + * @return + */ + @GET + @Path("getWorkSpaceStr") + public Response getWorkSpaceStr(@Context HttpServletRequest request) { + Workspace workspace = SSOHelper.getWorkspace(request); + Message message = Message.ok("").data("workspaceStr", DSSCommonUtils.COMMON_GSON.toJson(workspace)); + return Message.messageToResponse(message); + } + + /** + * 获取所有工程或者单个工程 + * + * @param request + * @param projectRequest + * @return + */ + @POST + @Path("getAllProjects") + public Response getAllProjects(@Context HttpServletRequest request, @Valid ProjectQueryRequest projectRequest) { + String username = SecurityFilter.getLoginUsername(request); + projectRequest.setUsername(username); + List dssProjectVos = projectService.getListByParam(projectRequest); + Message message = Message.ok("获取工作空间的工程成功").data("projects", dssProjectVos); + return Message.messageToResponse(message); + } + + /** + * 新建工程,通过和各个AppConn进行交互，将需要满足工程规范的所有的appconn进行创建工程 + */ + @POST + @Path("createProject") + public Response createProject(@Context HttpServletRequest request, @Valid ProjectCreateRequest projectCreateRequest) { + String username = SecurityFilter.getLoginUsername(request); + Workspace workspace = SSOHelper.getWorkspace(request); + try { + DSSProjectVo dssProjectVo = dssFrameworkProjectService.createProject(projectCreateRequest, username, workspace); + if (dssProjectVo != null) { + return Message.messageToResponse(Message.ok("创建工程成功").data("project", dssProjectVo)); + } else { + return Message.messageToResponse(Message.error("创建工程失败")); + } + } catch (final Throwable t) { + LOGGER.error("failed to create project {} for user {}", projectCreateRequest.getName(), username, t); + return RestfulUtils.dealError("创建工程失败:" + t.getMessage()); + } + } + + /** + * 编辑工程 + * + * @param request + * @param projectModifyRequest + * @return + */ + @POST + @Path("modifyProject") + public Response modifyProject(@Context HttpServletRequest request, @Valid ProjectModifyRequest projectModifyRequest) { + String username = SecurityFilter.getLoginUsername(request); + try { + dssFrameworkProjectService.modifyProject(projectModifyRequest, username); + return Message.messageToResponse(Message.ok("修改工程成功")); + } catch (Exception e) { + LOGGER.error("Failed to modify project {} for user {}", projectModifyRequest.getName(), username, e); + return RestfulUtils.dealError("修改工程失败:" + e.getMessage()); + } + } + + /** + * 删除工程 + * + * @param request + * @param projectDeleteRequest + * @return + */ + @POST + @Path("deleteProject") + public Response deleteProject(@Context HttpServletRequest request, @Valid ProjectDeleteRequest projectDeleteRequest) { + String username = SecurityFilter.getLoginUsername(request); + Workspace workspace = SSOHelper.getWorkspace(request); + try{ + projectService.deleteProject(username, projectDeleteRequest, workspace); + return RestfulUtils.dealOk("删除工程成功"); + }catch(final Throwable t){ + LOGGER.error("Failed to delete {} for user {}", projectDeleteRequest, username); + return RestfulUtils.dealError("删除工程失败"); + } + } + + @GET + @Path("/listApplicationAreas") + public Response listApplicationAreas(@Context HttpServletRequest req) { + String header = req.getHeader("Content-language").trim(); + ApplicationArea[] applicationAreas = ApplicationArea.values(); + List areas = new ArrayList<>(); + Arrays.stream(applicationAreas).forEach(item -> { + if ("zh-CN".equals(header)) { + areas.add(item.getName()); + } else { + areas.add(item.getEnName()); + } + }); + return Message.messageToResponse(Message.ok().data("applicationAreas", areas)); + } + + + @GET + @Path("/getProjectAbilities") + public Response getProjectAbilities(@Context HttpServletRequest request){ + //为了获取到此环境的能力，导入 导出 发布等 + String username = SecurityFilter.getLoginUsername(request); + try{ + List projectAbilities = projectService.getProjectAbilities(username); + return RestfulUtils.dealOk("获取工程能力成功", new Pair<>("projectAbilities", projectAbilities)); + }catch(final Throwable t){ + LOGGER.error("failed to get project ability for user {}", username, t); + return RestfulUtils.dealError("获取工程能力失败"); + } + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSFrameworkOrchestratorService.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSFrameworkOrchestratorService.java new file mode 100644 index 000000000..a9618f455 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSFrameworkOrchestratorService.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service; + +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorDeleteRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.vo.CommonOrchestratorVo; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; + + +public interface DSSFrameworkOrchestratorService { + + + + CommonOrchestratorVo createOrchestrator(String username, OrchestratorCreateRequest orchestratorCreateRequest, Workspace workspace) throws Exception; + + + + CommonOrchestratorVo modifyOrchestrator(String username, OrchestratorModifyRequest orchestratorModifyRequest, Workspace workspace) throws Exception; + + + + CommonOrchestratorVo deleteOrchestrator(String username, OrchestratorDeleteRequest orchestratorDeleteRequest, Workspace workspace) throws Exception; + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSFrameworkProjectService.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSFrameworkProjectService.java new file mode 100644 index 000000000..920e4ff41 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSFrameworkProjectService.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service; + +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.vo.DSSProjectDetailVo; +import com.webank.wedatasphere.dss.framework.project.entity.vo.DSSProjectVo; +import com.webank.wedatasphere.dss.framework.project.entity.vo.ProcessNode; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; + +import java.util.List; + + +public interface DSSFrameworkProjectService { + + DSSProjectDetailVo getProjectSettings(Long projectId); + + + DSSProjectVo createProject(ProjectCreateRequest projectCreateRequest, String username, Workspace workspace) throws Exception; + + void modifyProject(ProjectModifyRequest projectModifyRequest, String username) throws Exception; + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSOrchestratorService.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSOrchestratorService.java new file mode 100644 index 000000000..b397e358c --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSOrchestratorService.java @@ -0,0 +1,55 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service; + +import com.baomidou.mybatisplus.extension.service.IService; +import java.util.*; + +import com.webank.wedatasphere.dss.framework.common.exception.DSSFrameworkErrorException; +import com.webank.wedatasphere.dss.framework.project.entity.*; +import com.webank.wedatasphere.dss.framework.project.entity.request.*; +import com.webank.wedatasphere.dss.framework.project.entity.vo.OrchestratorBaseInfo; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestProjectImportOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateResponseRef; + + +public interface DSSOrchestratorService extends IService { + + + void saveOrchestrator(OrchestratorCreateRequest orchestratorCreateRequest, OrchestratorCreateResponseRef responseRef, String username) throws DSSFrameworkErrorException, DSSProjectErrorException; + + + void updateOrchestrator(OrchestratorModifyRequest orchestratorModifyRequest, String username) throws DSSFrameworkErrorException, DSSProjectErrorException; + + + List getListByPage(OrchestratorRequest orchestratorRequest, String username); + + + boolean deleteOrchestrator(OrchestratorDeleteRequest orchestratorDeleteRequest, String username) throws DSSProjectErrorException; + + + void isExistSameNameBeforeCreate(Long workspaceId, Long projectId, String orchestratorName) throws DSSFrameworkErrorException; + + + Long isExistSameNameBeforeUpdate(OrchestratorModifyRequest orchestratorModifRequest)throws DSSFrameworkErrorException; + + + DSSOrchestrator getOrchestratorById(Long id); + + Long importOrchestrator(RequestProjectImportOrchestrator orchestratorInfo) throws Exception; +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSProjectService.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSProjectService.java new file mode 100644 index 000000000..6d1d02d84 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSProjectService.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service; + +import com.baomidou.mybatisplus.extension.service.IService; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectDeleteRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectQueryRequest; +import com.webank.wedatasphere.dss.framework.project.entity.response.ProjectResponse; +import com.webank.wedatasphere.dss.framework.project.entity.vo.ProjectInfoVo; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestProjectImportOrchestrator; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; + +import java.util.List; +import java.util.Map; + +public interface DSSProjectService extends IService { + + + DSSProjectDO createProject(String username, ProjectCreateRequest projectCreateRequest); + + + void modifyProject(String username, ProjectModifyRequest modifyRequest) throws DSSProjectErrorException; + + + DSSProjectDO getProjectByName(String name); + + + DSSProjectDO getProjectById(Long id); + + + List getListByParam(ProjectQueryRequest projectRequest); + + + ProjectInfoVo getProjectInfoById(Long id); + + void saveProjectRelation(DSSProjectDO project, Map projectMap); + + Long getAppConnProjectId(Long dssProjectId, String appConnName, List dssLabels) throws Exception; + + void deleteProject(String username, ProjectDeleteRequest projectDeleteRequest, Workspace workspace) throws Exception; + + List getProjectAbilities(String username); + + + Long importOrchestrator(RequestProjectImportOrchestrator orchestratorInfo) throws Exception; +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSProjectUserService.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSProjectUserService.java new file mode 100644 index 000000000..65b3b4e75 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/DSSProjectUserService.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service; + +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectUser; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; + +import java.util.List; + +public interface DSSProjectUserService { + + + boolean isEditProjectAuth(Long projectId,String username) throws DSSProjectErrorException; + + + List getEditProjectList(Long projectId, String username); + + + void saveProjectUser(Long projectID, String username, ProjectCreateRequest dssProjectCreateRequest)throws Exception; + + + void modifyProjectUser(DSSProjectDO dbProject, ProjectModifyRequest projectModifyRequest, String loginuser)throws Exception; + + + List getListByParam(Long workspaceId, String username); + + + boolean isAdminByUsername(Long workspaceId,String username); +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSFrameworkOrchestratorServiceImpl.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSFrameworkOrchestratorServiceImpl.java new file mode 100644 index 000000000..54642b958 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSFrameworkOrchestratorServiceImpl.java @@ -0,0 +1,295 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service.impl; + + +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.framework.project.conf.ProjectConf; +import com.webank.wedatasphere.dss.framework.project.contant.OrchestratorTypeEnum; +import com.webank.wedatasphere.dss.framework.project.contant.ProjectServerResponse; +import com.webank.wedatasphere.dss.framework.project.dao.DSSOrchestratorMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSOrchestrator; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorDeleteRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.vo.CommonOrchestratorVo; +import com.webank.wedatasphere.dss.framework.project.entity.vo.ProjectInfoVo; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.framework.project.service.DSSFrameworkOrchestratorService; +import com.webank.wedatasphere.dss.framework.project.service.DSSOrchestratorService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.ref.DefaultOrchestratorCreateRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateResponseRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorFrameworkAppConn; +import com.webank.wedatasphere.dss.orchestrator.common.ref.impl.WorkflowOrchestratoDeleteRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.impl.WorkflowOrchestratorUpdateRequestRef; +import com.webank.wedatasphere.dss.orchestrator.core.type.OrchestratorKindEnum; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.service.RefCRUDService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import java.util.Arrays; +import java.util.Date; +import java.util.List; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +public class DSSFrameworkOrchestratorServiceImpl implements DSSFrameworkOrchestratorService { + + protected final Logger LOGGER = LoggerFactory.getLogger(getClass()); + + @Autowired + private DSSOrchestratorMapper orchestratorMapper; + @Autowired + private DSSOrchestratorService orchestratorService; + @Autowired + private DSSProjectService projectService; + + /** + * 1.拿到的dss orchestrator的appconn + * 2.然后创建 + * + * @param orchestratorCreateRequest 创建参数 + * @return + */ + @Override + @SuppressWarnings("ConstantConditions") + public CommonOrchestratorVo createOrchestrator(String username, OrchestratorCreateRequest orchestratorCreateRequest, + Workspace workspace) throws Exception { + //是否存在相同的编排名称 + orchestratorService.isExistSameNameBeforeCreate(orchestratorCreateRequest.getProjectId(),orchestratorCreateRequest.getWorkspaceId(),orchestratorCreateRequest.getOrchestratorName()); + //判断工程是否存在,并且取出工程名称和空间名称 + ProjectInfoVo projectInfoVo = projectService.getProjectInfoById(orchestratorCreateRequest.getProjectId()); + if (projectInfoVo == null) { + DSSExceptionUtils.dealErrorException(ProjectServerResponse.PROJECT_NOT_EXIST.getCode(), + ProjectServerResponse.PROJECT_NOT_EXIST.getMsg(), DSSProjectErrorException.class); + } + //1.去orchestratorFramework创建编排模式 + //2.将工程和orchestrator的关系存储到的数据库中 + LOGGER.info("{} begins to create a orchestrator {}", username, orchestratorCreateRequest); + //新建编排模式需要将编排模式的类型等内容提交给OrchestratorFramework + CommonOrchestratorVo orchestratorVo = new CommonOrchestratorVo(); + try { + OrchestratorKindEnum orchestratorKindEnum = OrchestratorKindEnum. + getType(OrchestratorTypeEnum.getTypeByKey(orchestratorCreateRequest.getOrchestratorMode())); + List dssLabels = Arrays.asList(new EnvDSSLabel(orchestratorCreateRequest.getLabels().getRoute())); + OrchestratorCreateRequestRef orchestratorCreateRequestRef = null; + String appconnName = "workflow"; + switch (orchestratorKindEnum) { + case WORKFLOW: + orchestratorCreateRequestRef = new DefaultOrchestratorCreateRequestRef(); + appconnName = "workflow"; + break; + case COMBINED: + appconnName = "combined"; + break; + case SINGLE_TASK: + appconnName = "singleTask"; + break; + default: + orchestratorCreateRequestRef = new DefaultOrchestratorCreateRequestRef(); + } + orchestratorCreateRequestRef.setUserName(username); + orchestratorCreateRequestRef.setDSSLabels(dssLabels); + orchestratorCreateRequestRef.setProjectId(orchestratorCreateRequest.getProjectId()); + orchestratorCreateRequestRef.setProjectName(StringUtils.isBlank(orchestratorCreateRequest.getProjectName()) ? projectInfoVo.getProjectName() : orchestratorCreateRequest.getProjectName()); + orchestratorCreateRequestRef.setWorkspaceName(StringUtils.isBlank(orchestratorCreateRequest.getWorkspaceName()) ? projectInfoVo.getWorkspaceName() : orchestratorCreateRequest.getWorkspaceName()); + DSSOrchestratorInfo dssOrchestratorInfo = new DSSOrchestratorInfo(); + dssOrchestratorInfo.setType(orchestratorKindEnum.getName()); + dssOrchestratorInfo.setDesc(orchestratorCreateRequest.getDescription()); + dssOrchestratorInfo.setCreateTime(new Date(System.currentTimeMillis())); + dssOrchestratorInfo.setAppConnName(appconnName); + dssOrchestratorInfo.setName(orchestratorCreateRequest.getOrchestratorName()); + dssOrchestratorInfo.setCreator(username); + dssOrchestratorInfo.setProjectId(orchestratorCreateRequest.getProjectId()); + dssOrchestratorInfo.setComment(orchestratorCreateRequest.getDescription()); + dssOrchestratorInfo.setSecondaryType(orchestratorCreateRequest.getOrchestratorWays().toString()); + orchestratorCreateRequestRef.setDssOrchestratorInfo(dssOrchestratorInfo); + //根据label获取对应label的创建operation + RefCreationOperation refCreationOperation = + getRefCreationOperation(dssLabels).getRefCreationOperation(); + //调用orchestrator服务 创建编排模式、编排模式版本 + OrchestratorCreateResponseRef responseRef =(OrchestratorCreateResponseRef) refCreationOperation.createRef(orchestratorCreateRequestRef); + LOGGER.info("operation ends to create ref, content is {}", responseRef.getContent()); + Long orchestratorId = responseRef.getOrcId(); + Long versionId = responseRef.getOrchestratorVersionId(); + LOGGER.info("orchestratorId is {}, and version is {}", orchestratorId, versionId); + orchestratorVo.setOrchestratorId(orchestratorId); + //保存编排模式到数据库 + orchestratorService.saveOrchestrator(orchestratorCreateRequest, responseRef, username); + } catch (final Exception e) { + DSSExceptionUtils.dealErrorException(60033, "failed to create orchestrator", e, DSSProjectErrorException.class); + } + return orchestratorVo; + } + + @Override + public CommonOrchestratorVo modifyOrchestrator(String username, OrchestratorModifyRequest orchestratorModifyRequest, Workspace workspace) throws Exception { + //判断工程是否存在,并且取出工程名称和空间名称 + ProjectInfoVo projectInfoVo = projectService.getProjectInfoById(orchestratorModifyRequest.getProjectId()); + if (projectInfoVo == null) { + DSSExceptionUtils.dealErrorException(ProjectServerResponse.PROJECT_NOT_EXIST.getCode(), + ProjectServerResponse.PROJECT_NOT_EXIST.getMsg(), DSSProjectErrorException.class); + } + //是否存在相同的编排名称 + Long orchestratorId = orchestratorService.isExistSameNameBeforeUpdate(orchestratorModifyRequest); + //1.去orchestratorFramework创建编排模式 + //2.将工程和orchestrator的关系存储到的数据库中 + LOGGER.info("{} begins to update a orchestrator {}", username, orchestratorModifyRequest); + //新建编排模式需要将编排模式的类型等内容提交给OrchestratorFramework + CommonOrchestratorVo orchestratorVo = new CommonOrchestratorVo(); + try { + OrchestratorKindEnum orchestratorKindEnum = OrchestratorKindEnum. + getType(OrchestratorTypeEnum.getTypeByKey(orchestratorModifyRequest.getOrchestratorMode())); + List dssLabels = Arrays.asList(new EnvDSSLabel(orchestratorModifyRequest.getLabels().getRoute())); + String appconnName = "workflow"; + WorkflowOrchestratorUpdateRequestRef orchestratorUpdateRequestRef = new WorkflowOrchestratorUpdateRequestRef(); + switch (orchestratorKindEnum) { + case COMBINED: + appconnName = "combined"; + break; + case SINGLE_TASK: + appconnName = "singleTask"; + break; + default: + appconnName = "workflow"; + orchestratorUpdateRequestRef = new WorkflowOrchestratorUpdateRequestRef(); + } + DSSOrchestratorInfo dssOrchestratorInfo = new DSSOrchestratorInfo(); + dssOrchestratorInfo.setId(orchestratorId); + dssOrchestratorInfo.setType(orchestratorKindEnum.getName()); + dssOrchestratorInfo.setDesc(orchestratorModifyRequest.getDescription()); + dssOrchestratorInfo.setCreateTime(new Date(System.currentTimeMillis())); + dssOrchestratorInfo.setAppConnName(appconnName); + dssOrchestratorInfo.setName(orchestratorModifyRequest.getOrchestratorName()); + dssOrchestratorInfo.setCreator(username); + dssOrchestratorInfo.setProjectId(orchestratorModifyRequest.getProjectId()); + dssOrchestratorInfo.setComment(orchestratorModifyRequest.getDescription()); + dssOrchestratorInfo.setSecondaryType(orchestratorModifyRequest.getOrchestratorWays().toString()); + String workspaceName = projectInfoVo.getWorkspaceName(); + + orchestratorUpdateRequestRef.setUserName(username); + orchestratorUpdateRequestRef.setWorkspaceName(workspaceName); + orchestratorUpdateRequestRef.setDssOrchestratorInfo(dssOrchestratorInfo); + orchestratorUpdateRequestRef.setDSSLabels(dssLabels); + + //根据label获取对应label的创建operation + RefUpdateOperation updateOperation = getRefCreationOperation(dssLabels).getRefUpdateOperation(); + //调用orchestrator服务 更新编排模式、编排模式版本 + LOGGER.info("----orchestratorUpdateRequestRef---> {}", orchestratorUpdateRequestRef); + CommonResponseRef responseRef = (CommonResponseRef)updateOperation.updateRef(orchestratorUpdateRequestRef); + LOGGER.info("operation ends to updateOrchestrator ref, UpdateResult is {},ErrorMsg is {}", responseRef.getResult(),responseRef.getErrorMsg()); + + //保存编排模式 + orchestratorService.updateOrchestrator(orchestratorModifyRequest, username); + } catch (final Exception e) { + DSSExceptionUtils.dealErrorException(60034, "failed to create orchestrator", e, DSSProjectErrorException.class); + } + return orchestratorVo; + } + + @Override + public CommonOrchestratorVo deleteOrchestrator(String username, OrchestratorDeleteRequest orchestratorDeleteRequest, Workspace workspace) throws Exception { + //判断工程是否存在,并且取出工程名称和空间名称 + ProjectInfoVo projectInfoVo = projectService.getProjectInfoById(orchestratorDeleteRequest.getProjectId()); + if (projectInfoVo == null) { + DSSExceptionUtils.dealErrorException(ProjectServerResponse.PROJECT_NOT_EXIST.getCode(), + ProjectServerResponse.PROJECT_NOT_EXIST.getMsg(), DSSProjectErrorException.class); + } + + DSSOrchestrator orchestrator = orchestratorService.getOrchestratorById(orchestratorDeleteRequest.getId()); + //1.去orchestratorFramework创建编排模式 + //2.将工程和orchestrator的关系存储到的数据库中 + LOGGER.info("{} begins to delete a orchestrator {}", username, orchestratorDeleteRequest); + //新建编排模式需要将编排模式的类型等内容提交给OrchestratorFramework + CommonOrchestratorVo orchestratorVo = new CommonOrchestratorVo(); + try { + String uuid = orchestratorMapper.getUUID(orchestrator.getOrchestratorId()); + if(StringUtils.isBlank(uuid)){ + DSSExceptionUtils.dealErrorException(60035,"工作流不存在", DSSProjectErrorException.class); + } + List dssLabels = Arrays.asList(new EnvDSSLabel(orchestratorDeleteRequest.getLabels().getRoute())); + WorkflowOrchestratoDeleteRequestRef orchestratorDeleteRequestRef = new WorkflowOrchestratoDeleteRequestRef(); + String workspaceName = projectInfoVo.getWorkspaceName(); + String projectName = projectInfoVo.getProjectName(); + orchestratorDeleteRequestRef.setUserName(username); + orchestratorDeleteRequestRef.setWorkspaceName(workspaceName); + orchestratorDeleteRequestRef.setProjectName(projectName); + orchestratorDeleteRequestRef.setDSSLabels(dssLabels); + orchestratorDeleteRequestRef.setOrcId(orchestrator.getOrchestratorId()); + RefDeletionOperation refDeletionOperation = + getRefCreationOperation(dssLabels).getRefDeletionOperation(); + + deleteOperation(refDeletionOperation, orchestratorDeleteRequestRef, uuid); + + //删除编排模式 + orchestratorService.deleteOrchestrator(orchestratorDeleteRequest, username); + } catch (final Exception e) { + DSSExceptionUtils.dealErrorException(60035, "failed to delete orchestrator", e, DSSProjectErrorException.class); + } + return orchestratorVo; + } + + protected void deleteOperation(RefDeletionOperation refDeletionOperation, + WorkflowOrchestratoDeleteRequestRef orchestratorDeleteRequestRef, String uuid) throws ExternalOperationFailedException { + //调用orchestrator服务 创建编排模式、编排模式版本 + refDeletionOperation.deleteRef(orchestratorDeleteRequestRef); + LOGGER.info("Operation deleteOrchestrator[DEV] for uuid {} success.", uuid); + } + + /** + * 获取更新的operation + * + * @param dssLabels 根据label获取对应label的创建operation + * todo 目前已经写死 + * @return + * @throws Exception + */ + public RefCRUDService getRefCreationOperation(List dssLabels) throws Exception { + OrchestratorFrameworkAppConn orcAppConn = AppConnManager.getAppConnManager().getAppConn(OrchestratorFrameworkAppConn.class); + + if (orcAppConn == null) { + LOGGER.error("orcAppConn is null, will not go on creating orchestrator"); + DSSExceptionUtils.dealErrorException(60028, "orcAppConn is null", DSSProjectErrorException.class); + } + + DevelopmentIntegrationStandard standard = ((OnlyDevelopmentAppConn)orcAppConn).getOrCreateDevelopmentStandard(); + if (standard == null) { + LOGGER.error("standard is null, will not go on creating orchestrator"); + DSSExceptionUtils.dealErrorException(60028, "standard is null", DSSProjectErrorException.class); + } + AppInstance appInstance = orcAppConn.getAppDesc().getAppInstancesByLabels(dssLabels).get(0); + + RefCRUDService service = standard.getRefCRUDService(appInstance); + return service; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSFrameworkProjectServiceImpl.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSFrameworkProjectServiceImpl.java new file mode 100644 index 000000000..fd1f2b6bf --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSFrameworkProjectServiceImpl.java @@ -0,0 +1,226 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service.impl; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyStructureAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.framework.project.contant.ProjectServerResponse; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.vo.DSSProjectDetailVo; +import com.webank.wedatasphere.dss.framework.project.entity.vo.DSSProjectVo; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.framework.project.exception.LambdaWarnException; +import com.webank.wedatasphere.dss.framework.project.service.DSSFrameworkProjectService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectUserService; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.structure.StructureIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectCreationOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRefImpl; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectResponseRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectUpdateOperation; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; +import java.util.HashMap; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; +import org.springframework.transaction.annotation.Transactional; + + +@Component +public class DSSFrameworkProjectServiceImpl implements DSSFrameworkProjectService { + private static final Logger LOGGER = LoggerFactory.getLogger(DSSFrameworkProjectServiceImpl.class); + @Autowired + private DSSProjectService projectService; + @Autowired + private DSSProjectUserService projectUserService; + + public static final String MODE_SPLIT = ","; + + private static final boolean STRICT_PROJECT_CREATE_MODE = CommonVars.apply("wds.dss.project.strict.mode", false).getValue(); + + @Override + public DSSProjectDetailVo getProjectSettings(Long projectId) { + return null; + } + + @Override + @Transactional(rollbackFor = Exception.class) + public DSSProjectVo createProject(ProjectCreateRequest projectCreateRequest, String username, Workspace workspace) throws Exception { + //1.新建DSS工程,这样才能进行回滚,如果后面去DSS工程，可能会由于DSS工程建立失败了，但是仍然无法去回滚第三方系统的工程 + //2.开始创建appconn的相关的工程，如果失败了，抛异常，然后进行数据库进行回滚 + + //判断工程是否存在相同的名称 + DSSProjectDO dbProject = projectService.getProjectByName(projectCreateRequest.getName()); + if (dbProject != null) { + DSSExceptionUtils.dealErrorException(60022, String.format("project name already has the same name %s ", projectCreateRequest.getName()), DSSProjectErrorException.class); + } + //todo 创建appconn的相关的工程 还没有调试通过 + Map projectMap = createAppConnProject(projectCreateRequest, username, workspace); + if (null == projectMap) { + LOGGER.error("projectMap is null, it means some appConns create project failed"); + throw new DSSProjectErrorException(71000, "projectMap is null, create project in appconn failed"); + } + //3.保存dss_project + DSSProjectDO project = projectService.createProject(username, projectCreateRequest); + //4.保存dss_project_user 工程与用户关系 + projectUserService.saveProjectUser(project.getId(), username, projectCreateRequest); + //5.保存dss工程与其他工程的对应关系,应该都是以id来作为标识 + if (null != projectMap && projectMap.size() > 0) { + projectService.saveProjectRelation(project, projectMap); + } + DSSProjectVo dssProjectVo = new DSSProjectVo(); + dssProjectVo.setDescription(project.getDescription()); + dssProjectVo.setId(project.getId()); + dssProjectVo.setName(project.getName()); + return dssProjectVo; + } + + + @Override + public void modifyProject(ProjectModifyRequest projectModifyRequest, String username) throws Exception { + DSSProjectDO dbProject = projectService.getProjectById(projectModifyRequest.getId()); + //如果不是工程的创建人，则校验是否管理员 + if (!username.equalsIgnoreCase(dbProject.getCreateBy())) { + boolean isAdmin = projectUserService.isAdminByUsername(projectModifyRequest.getWorkspaceId(), username); + //非管理员 + if (!isAdmin) { + DSSExceptionUtils.dealErrorException(ProjectServerResponse.PROJECT_IS_NOT_ADMIN.getCode(), ProjectServerResponse.PROJECT_IS_NOT_ADMIN.getMsg(), DSSProjectErrorException.class); + } + } + //工程不存在 + if (dbProject == null) { + LOGGER.error("{} project id is null, can not modify", projectModifyRequest.getName()); + DSSExceptionUtils.dealErrorException(60021, + String.format("%s project id is null, can not modify", projectModifyRequest.getName()), DSSProjectErrorException.class); + } + //不允许修改工程名称 + if (!dbProject.getName().toUpperCase().equals(projectModifyRequest.getName().toUpperCase())) { + DSSExceptionUtils.dealErrorException(ProjectServerResponse.PROJECT_NOT_EDIT_NAME.getCode(), ProjectServerResponse.PROJECT_NOT_EDIT_NAME.getMsg(), DSSProjectErrorException.class); + } + //1.统一修改各个接入的第三方的系统的工程状态信息 + //todo 第三方的工程修改接口还是没有调试通过 +// modifyThirdProject(projectModifyRequest, username); + //2.修改dss_project DSS基本工程信息 + projectService.modifyProject(username, projectModifyRequest); + try { + //todo 3.修改dss_project_user 工程与用户关系 这一步还没有调试通过 + projectUserService.modifyProjectUser(dbProject, projectModifyRequest, username); + } catch (Exception e) { + LOGGER.error("modifyProjectUserError:", e); + } + } + + //统一修改各个接入的第三方的系统的工程状态信息 修改dss_project调用 + private void modifyThirdProject(ProjectModifyRequest projectModifyRequest, String username) { + for (AppConn appConn : AppConnManager.getAppConnManager().listAppConns()) { + ProjectRequestRef projectRequestRef = new ProjectRequestRefImpl(); + projectRequestRef.setDescription(projectModifyRequest.getDescription()); + projectRequestRef.setUpdateBy(username); + projectRequestRef.setName(projectModifyRequest.getName()); + if (appConn instanceof OnlyStructureAppConn) { + StructureIntegrationStandard appStandard = ((OnlyStructureAppConn) appConn).getOrCreateStructureStandard(); + for (AppInstance appInstance : appConn.getAppDesc().getAppInstances()) { + ProjectService projectService = appStandard.getProjectService(appInstance); + ProjectUpdateOperation operation = projectService.getProjectUpdateOperation(); + try { + ProjectResponseRef responseRef = operation.updateProject(projectRequestRef); + } catch (ExternalOperationFailedException e) { + DSSExceptionUtils.dealWarnException(60015, + String.format("failed to update project %s", projectModifyRequest.getName()), e, LambdaWarnException.class); + } + } + } + } + } + + //1.新建DSS工程,这样才能进行回滚,如果后面去DSS工程，可能会由于DSS工程建立失败了，但是仍然无法去回滚第三方系统的工程 新增dss_project调用 + private Map createAppConnProject(ProjectCreateRequest dssProjectCreateRequest, String username, + Workspace workspace) throws DSSProjectErrorException { + Map projectMap = new HashMap<>(16); + Map successAppConns = new HashMap<>(16); + boolean createFailed = false; + ProjectRequestRefImpl requestRef = new ProjectRequestRefImpl(); + requestRef.setName(dssProjectCreateRequest.getName()); + requestRef.setCreateBy(username); + requestRef.setDescription(dssProjectCreateRequest.getDescription()); + requestRef.setWorkspaceName(dssProjectCreateRequest.getWorkspaceName()); + requestRef.setWorkspace(workspace); + for (AppConn appConn : AppConnManager.getAppConnManager().listAppConns()) { + if (appConn instanceof OnlyStructureAppConn) { + OnlyStructureAppConn onlyStructureAppConn = (OnlyStructureAppConn) appConn; + StructureIntegrationStandard appStandard = onlyStructureAppConn.getOrCreateStructureStandard(); + //如果该AppConn是有structureIntegrationStandard的话,那么所有的appinstance都要进行新建工程 + for (AppInstance appInstance : appConn.getAppDesc().getAppInstances()) { + ProjectService projectService = appStandard.getProjectService(appInstance); + if(projectService == null) { + continue; + } + ProjectCreationOperation projectCreationOperation = projectService.getProjectCreationOperation(); + try { + LOGGER.info("Begin to create project {} in {}", dssProjectCreateRequest.getName(), appConn.getAppDesc().getAppName()); + ProjectResponseRef responseRef = projectCreationOperation.createProject(requestRef); + LOGGER.info("End to create project {} in {}, response projectId is {} ", + dssProjectCreateRequest.getName(), appConn.getAppDesc().getAppName(), responseRef.getProjectRefId()); + successAppConns.put(appConn, responseRef); + Long projectRefId = responseRef.getProjectRefId(); + projectMap.put(appInstance, projectRefId); + } catch (final Exception e) { + LOGGER.error("Failed to create project {} in {}", dssProjectCreateRequest.getName(), appConn.getAppDesc().getAppInstances(), e); + createFailed = true; + //break; + } + } + } + } + //如果创建失败并且是严格创建模式 + if (createFailed && STRICT_PROJECT_CREATE_MODE) { + //如果一个AppInstance实例是失败的，那么我们将所有已经建的工程给撤销掉 + for (AppConn appConn : successAppConns.keySet()) { + StructureIntegrationStandard appStandard = null; + if (appConn instanceof OnlyStructureAppConn) { + OnlyStructureAppConn onlyStructureAppConn = (OnlyStructureAppConn) appConn; + appStandard = onlyStructureAppConn.getOrCreateStructureStandard(); + for (AppInstance appInstance : appConn.getAppDesc().getAppInstances()) { + ProjectService projectService = appStandard.getProjectService(appInstance); + ProjectDeletionOperation operation = projectService.getProjectDeletionOperation(); + try { + operation.deleteProject(requestRef); + } catch (ExternalOperationFailedException e) { + LOGGER.error("Failed to delete project {} in {}", + requestRef.getName(), appConn.getAppDesc().getAppName(), e); + //TODO 如果删除不了，可以先不管 + } + } + } + } + projectMap = null; + } + return projectMap; + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSOrchestratorServiceImpl.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSOrchestratorServiceImpl.java new file mode 100644 index 000000000..d301a718f --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSOrchestratorServiceImpl.java @@ -0,0 +1,306 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service.impl; + +import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; +import com.baomidou.mybatisplus.core.conditions.update.UpdateWrapper; +import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl; +import com.webank.wedatasphere.dss.framework.common.exception.DSSFrameworkErrorException; +import com.webank.wedatasphere.dss.framework.project.contant.OrchestratorTypeEnum; +import com.webank.wedatasphere.dss.framework.project.contant.ProjectServerResponse; +import com.webank.wedatasphere.dss.framework.project.dao.DSSOrchestratorMapper; +import com.webank.wedatasphere.dss.framework.project.dao.DSSProjectMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSOrchestrator; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectUser; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorDeleteRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.OrchestratorRequest; +import com.webank.wedatasphere.dss.framework.project.entity.vo.OrchestratorBaseInfo; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.framework.project.service.DSSOrchestratorService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectUserService; +import com.webank.wedatasphere.dss.framework.project.utils.ProjectStringUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestProjectImportOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCreateResponseRef; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.BeanUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + + +@Service +public class DSSOrchestratorServiceImpl extends ServiceImpl implements DSSOrchestratorService { + private static final Logger LOGGER = LoggerFactory.getLogger(DSSOrchestratorServiceImpl.class); + public static final String MODE_SPLIT = ","; + @Autowired + private DSSOrchestratorMapper orchestratorMapper; + @Autowired + private DSSProjectUserService projectUserService; + @Autowired + private DSSProjectMapper dssProjectMapper; + + private final Sender orcSender = DSSSenderServiceFactory.getOrCreateServiceInstance().getScheduleOrcSender(); + + /** + * 保存编排模式 + * + * @param orchestratorCreateRequest + * @param responseRef + * @param username + * @throws DSSFrameworkErrorException + * @throws DSSProjectErrorException + */ + @Override + public void saveOrchestrator(OrchestratorCreateRequest orchestratorCreateRequest, OrchestratorCreateResponseRef responseRef, String username) throws DSSFrameworkErrorException, DSSProjectErrorException { + //todo 使用工程的权限关系来限定 校验当前登录用户是否含有修改权限 + projectUserService.isEditProjectAuth(orchestratorCreateRequest.getProjectId(), username); + DSSOrchestrator orchestrator = new DSSOrchestrator(); + orchestrator.setWorkspaceId(orchestratorCreateRequest.getWorkspaceId()); + orchestrator.setProjectId(orchestratorCreateRequest.getProjectId()); + //编排模式id（工作流,调用orchestrator服务返回的orchestratorId） + orchestrator.setOrchestratorId(responseRef.getOrcId()); + //编排模式版本id（工作流,调用orchestrator服务返回的orchestratorVersionId） + orchestrator.setOrchestratorVersionId(responseRef.getOrchestratorVersionId()); + //编排模式-名称 + orchestrator.setOrchestratorName(orchestratorCreateRequest.getOrchestratorName()); + //编排模式-类型 + orchestrator.setOrchestratorMode(orchestratorCreateRequest.getOrchestratorMode()); + //编排模式-方式 + orchestrator.setOrchestratorWay(ProjectStringUtils.getModeStr(orchestratorCreateRequest.getOrchestratorWays())); + //编排模式-用途 + orchestrator.setUses(orchestratorCreateRequest.getUses()); + //编排模式-描述 + orchestrator.setDescription(orchestratorCreateRequest.getDescription()); + //编排模式-创建人 + orchestrator.setCreateUser(username); + orchestrator.setCreateTime(new Date()); + orchestrator.setUpdateTime(new Date()); + this.save(orchestrator); + } + + /** + * 更新编排模式 + * + * @param orchestratorModifRequest + * @param username + * @throws DSSFrameworkErrorException + * @throws DSSProjectErrorException + */ + @Override + public void updateOrchestrator(OrchestratorModifyRequest orchestratorModifRequest, String username) throws DSSFrameworkErrorException, DSSProjectErrorException { + //todo 使用工程的权限关系来限定 校验当前登录用户是否含有修改权限 + projectUserService.isEditProjectAuth(orchestratorModifRequest.getProjectId(), username); + + DSSOrchestrator orchestrator = new DSSOrchestrator(); + orchestrator.setId(orchestratorModifRequest.getId()); + //编排模式-名称 + orchestrator.setOrchestratorName(orchestratorModifRequest.getOrchestratorName()); + //编排模式 + orchestrator.setOrchestratorMode(orchestratorModifRequest.getOrchestratorMode()); + //编排模式-方式 + orchestrator.setOrchestratorWay(ProjectStringUtils.getModeStr(orchestratorModifRequest.getOrchestratorWays())); + //编排模式-用途 + orchestrator.setUses(orchestratorModifRequest.getUses()); + //编排模式-描述 + orchestrator.setDescription(orchestratorModifRequest.getDescription()); + //编排模式-更新人 + orchestrator.setUpdateUser(username); + //编排模式-更新时间 + orchestrator.setUpdateTime(new Date()); + this.updateById(orchestrator); + } + + //新建前是否存在相同的编排名称 + @Override + public void isExistSameNameBeforeCreate(Long workspaceId, Long projectId, String arrangeName) throws DSSFrameworkErrorException { + QueryWrapper queryWrapper = new QueryWrapper(); + queryWrapper.eq("workspace_id", workspaceId); + queryWrapper.eq("project_id", projectId); + queryWrapper.eq("orchestrator_name", arrangeName); + List list = this.list(queryWrapper); + if (!CollectionUtils.isEmpty(list)) { + DSSFrameworkErrorException.dealErrorException(60000, "编排名称已经存在"); + } + } + + //是否存在相同的编排名称,如果不存在相同的编排名称則返回编排id + @Override + public Long isExistSameNameBeforeUpdate(OrchestratorModifyRequest orchestratorModifRequest) throws DSSFrameworkErrorException { + QueryWrapper queryWrapper = new QueryWrapper(); + queryWrapper.eq("workspace_id", orchestratorModifRequest.getWorkspaceId()); + queryWrapper.eq("project_id", orchestratorModifRequest.getProjectId()); + queryWrapper.eq("id", orchestratorModifRequest.getId()); + List list = this.list(queryWrapper); + if (CollectionUtils.isEmpty(list)) { + DSSFrameworkErrorException.dealErrorException(60000, "编排模式ID=" + orchestratorModifRequest.getId() + "不存在"); + } + //是否存在相同的编排名称 + if (!orchestratorModifRequest.getOrchestratorName().equals(list.get(0).getOrchestratorName())) { + isExistSameNameBeforeCreate(orchestratorModifRequest.getWorkspaceId(), orchestratorModifRequest.getProjectId(), orchestratorModifRequest.getOrchestratorName()); + } + return list.get(0).getOrchestratorId(); + } + + + /** + * 查询编排模式 + * + * @param orchestratorRequest + * @param username + * @return + */ + @Override + public List getListByPage(OrchestratorRequest orchestratorRequest, String username) { + QueryWrapper queryWrapper = new QueryWrapper(); + queryWrapper.eq("workspace_id", orchestratorRequest.getWorkspaceId()); + queryWrapper.eq("project_id", orchestratorRequest.getProjectId()); + if (StringUtils.isNotBlank(orchestratorRequest.getOrchestratorMode())) { + queryWrapper.eq("orchestrator_mode", orchestratorRequest.getOrchestratorMode()); + } + List list = this.list(queryWrapper); + List retList = new ArrayList(list.size()); + if (!CollectionUtils.isEmpty(list)) { + //获取工程的权限等级 + List projectUserList = projectUserService.getEditProjectList(orchestratorRequest.getProjectId(), username); + //获取最大权限 + + Integer priv = CollectionUtils.isEmpty(projectUserList) ? null : projectUserList.stream().mapToInt(DSSProjectUser::getPriv).max().getAsInt(); + OrchestratorBaseInfo orchestratorBaseInfo = null; + for (DSSOrchestrator orchestrator : list) { + orchestratorBaseInfo = new OrchestratorBaseInfo(); + BeanUtils.copyProperties(orchestrator, orchestratorBaseInfo); + orchestratorBaseInfo.setOrchestratorWays(ProjectStringUtils.convertList(orchestrator.getOrchestratorWay())); + orchestratorBaseInfo.setPriv(priv); + retList.add(orchestratorBaseInfo); + } + } + return retList; + } + + /** + * 删除编排模式 + * + * @param orchestratorDeleteRequest + * @param username + * @return + * @throws DSSProjectErrorException + */ + @Override + @SuppressWarnings("all") + public boolean deleteOrchestrator(OrchestratorDeleteRequest orchestratorDeleteRequest, String username) throws DSSProjectErrorException { + //todo 使用工程的权限关系来限定 校验当前登录用户是否含有修改权限 + projectUserService.isEditProjectAuth(orchestratorDeleteRequest.getProjectId(), username); + + UpdateWrapper updateWrapper = new UpdateWrapper(); + updateWrapper.eq("workspace_id", orchestratorDeleteRequest.getWorkspaceId()); + updateWrapper.eq("project_id", orchestratorDeleteRequest.getProjectId()); + updateWrapper.eq("id", orchestratorDeleteRequest.getId()); + return this.remove(updateWrapper); + } + + @Override + public DSSOrchestrator getOrchestratorById(Long id) { + return this.getById(id); + } + + @Override + public Long importOrchestrator(RequestProjectImportOrchestrator orchestratorInfo) throws Exception { + DSSProjectDO projectDO = dssProjectMapper.selectById(orchestratorInfo.getProjectId()); + if (projectDO == null) { + DSSFrameworkErrorException.dealErrorException(ProjectServerResponse.PROJECT_NOT_EXIST.getCode(), + ProjectServerResponse.PROJECT_NOT_EXIST.getMsg()); + } + //校验参数 + String type = orchestratorInfo.getType(); + if (StringUtils.isEmpty(type)) { + DSSFrameworkErrorException.dealErrorException(60000, "编排模式类型不能为不存在"); + } + Long orcId = orchestratorInfo.getId(); + String orchestratorMode = OrchestratorTypeEnum.getKeyByOrcType(type); + String orchestratorWay = orchestratorInfo.getSecondaryType(); + if (StringUtils.isNotBlank(orchestratorWay)) { + orchestratorWay = orchestratorWay.replace("[", ",").replace("]", ","); + } + //查询是否存在相同的编排模式 + QueryWrapper queryWrapper = new QueryWrapper(); + queryWrapper.eq("orchestrator_id", orcId); + queryWrapper.eq("project_id", projectDO.getId()); + List list = this.list(queryWrapper); + //如果存在即更新既可以 + if (!CollectionUtils.isEmpty(list)) { + //导入更新 + return updateOrcByImport(list.get(0), orchestratorMode,orchestratorWay,orchestratorInfo); + } else { + //导入新增 + return insertOrcByImport(projectDO,orchestratorMode,orchestratorWay,orchestratorInfo); + } + } + + //导入更新 + public Long updateOrcByImport(DSSOrchestrator dbEntity, String orchestratorMode, String orchestratorWay, RequestProjectImportOrchestrator orchestratorInfo) { + DSSOrchestrator updateEntity = new DSSOrchestrator(); + BeanUtils.copyProperties(dbEntity, updateEntity); + //更新字段 + updateEntity.setOrchestratorName(orchestratorInfo.getName()); + updateEntity.setOrchestratorMode(orchestratorMode); + updateEntity.setOrchestratorWay(orchestratorWay); + updateEntity.setDescription(orchestratorInfo.getDesc()); + updateEntity.setUpdateUser(orchestratorInfo.getCreator()); + updateEntity.setUpdateTime(new Date()); + //数据库更新 + this.updateById(updateEntity); + return 0L; + } + + //导入新增 + public Long insertOrcByImport(DSSProjectDO projectDO, String orchestratorMode, String orchestratorWay, RequestProjectImportOrchestrator orchestratorInfo) + throws DSSFrameworkErrorException { + //新增，查看是否含有同名的编排模式 + /* QueryWrapper queryNameWrapper = new QueryWrapper(); + queryNameWrapper.eq("workspace_id", projectDO.getWorkspaceId()); + queryNameWrapper.eq("project_id", orchestratorInfo.getProjectId()); + queryNameWrapper.eq("orchestrator_name", orchestratorInfo.getName()); + List sameNameList = this.list(queryNameWrapper); + if (!CollectionUtils.isEmpty(sameNameList)) { + DSSFrameworkErrorException.dealErrorException(60000, "编排名称已经存在"); + }*/ + DSSOrchestrator orchestrator = new DSSOrchestrator(); + orchestrator.setProjectId(projectDO.getId()); + orchestrator.setOrchestratorId(orchestratorInfo.getId()); + orchestrator.setOrchestratorVersionId(orchestratorInfo.getVersionId()); + orchestrator.setOrchestratorName(orchestratorInfo.getName()); + orchestrator.setOrchestratorMode(orchestratorMode); + orchestrator.setOrchestratorWay(orchestratorWay); + orchestrator.setDescription(orchestratorInfo.getDesc()); + orchestrator.setCreateUser(orchestratorInfo.getCreator()); + orchestrator.setCreateTime(new Date()); + orchestrator.setUpdateTime(new Date()); + orchestrator.setWorkspaceId(projectDO.getWorkspaceId()); + this.save(orchestrator); + return orchestrator.getId(); + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSProjectServiceImpl.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSProjectServiceImpl.java new file mode 100644 index 000000000..9a64e66f2 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSProjectServiceImpl.java @@ -0,0 +1,259 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service.impl; + +import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; +import com.baomidou.mybatisplus.core.conditions.update.UpdateWrapper; +import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl; +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyStructureAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.framework.project.conf.ProjectConf; +import com.webank.wedatasphere.dss.framework.project.contant.ProjectUserPrivEnum; +import com.webank.wedatasphere.dss.framework.project.dao.DSSProjectMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.po.ProjectRelationPo; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectDeleteRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectQueryRequest; +import com.webank.wedatasphere.dss.framework.project.entity.response.ProjectResponse; +import com.webank.wedatasphere.dss.framework.project.entity.vo.ProjectInfoVo; +import com.webank.wedatasphere.dss.framework.project.entity.vo.QueryProjectVo; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.framework.project.service.DSSOrchestratorService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectUserService; +import com.webank.wedatasphere.dss.framework.project.utils.ProjectStringUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestProjectImportOrchestrator; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRef; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectRequestRefImpl; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; + + +public class DSSProjectServiceImpl extends ServiceImpl implements DSSProjectService { + private static final Logger LOGGER = LoggerFactory.getLogger(DSSProjectServiceImpl.class); + @Autowired + private DSSProjectMapper projectMapper; + @Autowired + private DSSProjectUserService projectUserService; + @Autowired + private DSSOrchestratorService orchestratorService; + public static final String MODE_SPLIT = ","; + public static final String KEY_SPLIT = "-"; + private final String SUPPORT_ABILITY = ProjectConf.SUPPORT_ABILITY.getValue(); + + @Override + public DSSProjectDO createProject(String username, ProjectCreateRequest projectCreateRequest) { + DSSProjectDO project = new DSSProjectDO(); + project.setName(projectCreateRequest.getName()); + project.setWorkspaceId(projectCreateRequest.getWorkspaceId()); + project.setCreateBy(username); + project.setUsername(username); + project.setCreateTime(new Date()); + project.setBusiness(projectCreateRequest.getBusiness()); + project.setProduct(projectCreateRequest.getProduct()); + project.setUpdateTime(new Date()); + project.setDescription(projectCreateRequest.getDescription()); + project.setApplicationArea(projectCreateRequest.getApplicationArea()); + //开发流程，编排模式组拼接 前后进行英文逗号接口 + project.setDevProcess(ProjectStringUtils.getModeStr(projectCreateRequest.getDevProcessList())); + project.setOrchestratorMode(ProjectStringUtils.getModeStr(projectCreateRequest.getOrchestratorModeList())); + projectMapper.insert(project); + return project; + } + + //修改dss_project工程字段 + @Override + public void modifyProject(String username, ProjectModifyRequest projectModifyRequest) throws DSSProjectErrorException { + //校验当前登录用户是否含有修改权限 + projectUserService.isEditProjectAuth(projectModifyRequest.getId(), username); + DSSProjectDO project = new DSSProjectDO(); + //修改的字段 + project.setDescription(projectModifyRequest.getDescription()); + project.setUpdateTime(new Date()); + project.setUpdateByStr(username); + project.setDevProcess(ProjectStringUtils.getModeStr(projectModifyRequest.getDevProcessList())); + if (StringUtils.isNotBlank(projectModifyRequest.getApplicationArea())) { + project.setApplicationArea(Integer.valueOf(projectModifyRequest.getApplicationArea())); + } + project.setOrchestratorMode(ProjectStringUtils.getModeStr(projectModifyRequest.getOrchestratorModeList())); + project.setBusiness(projectModifyRequest.getBusiness()); + project.setProduct(projectModifyRequest.getProduct()); + + UpdateWrapper updateWrapper = new UpdateWrapper(); + updateWrapper.eq("id", projectModifyRequest.getId()); + updateWrapper.eq("workspace_id", projectModifyRequest.getWorkspaceId()); + projectMapper.update(project, updateWrapper); + } + + @Override + public DSSProjectDO getProjectByName(String name) { + QueryWrapper projectQueryWrapper = new QueryWrapper(); + projectQueryWrapper.eq("name", name); + List projectList = projectMapper.selectList(projectQueryWrapper); + return CollectionUtils.isEmpty(projectList) ? null : projectList.get(0); + } + + @Override + public DSSProjectDO getProjectById(Long id) { + return projectMapper.selectById(id); + } + + @Override + public List getListByParam(ProjectQueryRequest projectRequest) { + //根据dss_project、dss_project_user查询出所在空间登录用户相关的工程 + List list = projectMapper.getListByParam(projectRequest); + if (CollectionUtils.isEmpty(list)) { + return new ArrayList<>(); + } + + List projectResponseList = new ArrayList<>(); + ProjectResponse projectResponse = null; + for (QueryProjectVo projectVo : list) { + if (projectVo.getVisible() == 0) { + continue; + } + projectResponse = new ProjectResponse(); + projectResponse.setApplicationArea(projectVo.getApplicationArea()); + projectResponse.setId(projectVo.getId()); + projectResponse.setBusiness(projectVo.getBusiness()); + projectResponse.setCreateBy(projectVo.getCreateBy()); + projectResponse.setDescription(projectVo.getDescription()); + projectResponse.setName(projectVo.getName()); + projectResponse.setProduct(projectVo.getProduct()); + projectResponse.setSource(projectVo.getSource()); + projectResponse.setArchive(projectVo.getArchive()); + projectResponse.setCreateTime(projectVo.getCreateTime()); + projectResponse.setUpdateTime(projectVo.getUpdateTime()); + projectResponse.setDevProcessList(ProjectStringUtils.convertList(projectVo.getDevProcess())); + projectResponse.setOrchestratorModeList(ProjectStringUtils.convertList(projectVo.getOrchestratorMode())); + projectResponseList.add(projectResponse); + /** + * 拆分有projectId +"-" + priv + "-" + username的拼接而成的字段， + * 从而得到：查看权限用户、编辑权限用户、发布权限用户 + */ + String pusername = projectVo.getPusername(); + if (StringUtils.isEmpty(pusername)) { + continue; + } + Map> userPricMap = new HashMap<>(); + String[] tempstrArr = pusername.split(MODE_SPLIT); + + for (String s : tempstrArr) { + String[] strArr = s.split(KEY_SPLIT); + String key = strArr[0] + KEY_SPLIT + strArr[1]; + userPricMap.computeIfAbsent(key, k -> new ArrayList<>()); + userPricMap.get(key).add(strArr[2]); + } + List accessUsers = userPricMap.get(projectVo.getId() + KEY_SPLIT + ProjectUserPrivEnum.PRIV_ACCESS.getRank()); + List editUsers = userPricMap.get(projectVo.getId() + KEY_SPLIT + ProjectUserPrivEnum.PRIV_EDIT.getRank()); + List releaseUsers = userPricMap.get(projectVo.getId() + KEY_SPLIT + ProjectUserPrivEnum.PRIV_RELEASE.getRank()); + projectResponse.setAccessUsers(CollectionUtils.isEmpty(accessUsers) ? new ArrayList<>() : accessUsers.stream().distinct().collect(Collectors.toList())); + projectResponse.setEditUsers(CollectionUtils.isEmpty(editUsers) ? new ArrayList<>() : editUsers.stream().distinct().collect(Collectors.toList())); + projectResponse.setReleaseUsers(CollectionUtils.isEmpty(releaseUsers) ? new ArrayList<>() : releaseUsers.stream().distinct().collect(Collectors.toList())); + } + return projectResponseList; + } + + @Override + public ProjectInfoVo getProjectInfoById(Long id) { + return projectMapper.getProjectInfoById(id); + } + + + @Override + public void saveProjectRelation(DSSProjectDO project, Map projectMap) { + List relationPos = new ArrayList<>(); + projectMap.forEach((k, v) -> { + relationPos.add(new ProjectRelationPo(project.getId(), k.getId(), v)); + }); + projectMapper.saveProjectRelation(relationPos); + } + + + @Override + public Long getAppConnProjectId(Long dssProjectId, String appConnName, List dssLabels) throws Exception { + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(appConnName); + List appInstances = appConn.getAppDesc().getAppInstancesByLabels(dssLabels); + if (appInstances.get(0) != null) { + Long appInstanceId = appInstances.get(0).getId(); + return projectMapper.getAppConnProjectId(appInstanceId, dssProjectId); + } else { + LOGGER.error("appInstances is null {}", appInstances); + return null; + } + } + + @Override + public void deleteProject(String username, ProjectDeleteRequest projectDeleteRequest, Workspace workspace) throws Exception { + LOGGER.warn("user {} begins to delete project {}.", username, projectDeleteRequest); + DSSProjectDO DSSProjectDO = projectMapper.selectById(projectDeleteRequest.getId()); + if (DSSProjectDO == null) { + throw new DSSErrorException(600001, "工程不存在" ); + } + if(projectDeleteRequest.isIfDelOtherSys()) { + ProjectRequestRef projectRequestRef = new ProjectRequestRefImpl(); + projectRequestRef.setName(DSSProjectDO.getName()); + projectRequestRef.setWorkspace(workspace); + AppConnManager.getAppConnManager().listAppConns().stream().filter(appConn -> appConn instanceof OnlyStructureAppConn).forEach(appConn -> { + OnlyStructureAppConn structureAppConn = (OnlyStructureAppConn) appConn; + appConn.getAppDesc().getAppInstances().forEach(DSSExceptionUtils.handling(appInstance -> { + ProjectService projectService = structureAppConn.getOrCreateStructureStandard().getProjectService(appInstance); + if(projectService != null) { + ProjectDeletionOperation projectDeletionOperation = projectService.getProjectDeletionOperation(); + if(projectDeletionOperation != null) { + projectDeletionOperation.deleteProject(projectRequestRef); + } + } + })); + }); + } + projectMapper.deleteProject(projectDeleteRequest.getId()); + LOGGER.warn("User {} deleted project {}.", username, DSSProjectDO.getName()); + } + + @Override + public List getProjectAbilities(String username) { + LOGGER.info("{} begins to get project ability", username); + return Arrays.asList(SUPPORT_ABILITY.trim().split(",")); + } + + @Override + public Long importOrchestrator(RequestProjectImportOrchestrator orchestratorInfo) throws Exception { + return orchestratorService.importOrchestrator(orchestratorInfo); + } + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSProjectUserServiceImpl.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSProjectUserServiceImpl.java new file mode 100644 index 000000000..b3103736e --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/service/impl/DSSProjectUserServiceImpl.java @@ -0,0 +1,188 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.service.impl; + +import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.framework.project.contant.ProjectServerResponse; +import com.webank.wedatasphere.dss.framework.project.contant.ProjectUserPrivEnum; +import com.webank.wedatasphere.dss.framework.project.dao.DSSProjectUserMapper; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO; +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectUser; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectCreateRequest; +import com.webank.wedatasphere.dss.framework.project.entity.request.ProjectModifyRequest; +import com.webank.wedatasphere.dss.framework.project.exception.DSSProjectErrorException; +import com.webank.wedatasphere.dss.framework.project.server.service.BMLService; +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectUserService; +import com.webank.wedatasphere.dss.framework.project.utils.ProjectUserUtils; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; +import org.apache.commons.collections.CollectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Qualifier; + + +public class DSSProjectUserServiceImpl implements DSSProjectUserService { + + protected final Logger LOGGER = LoggerFactory.getLogger(getClass()); + + @Autowired + private DSSProjectUserMapper projectUserMapper; + @Autowired + @Qualifier("projectServerBMLService") + private BMLService bmlService; + + /** + * 是否有修改工程权限 + * @param projectId + * @param username + * @return + */ + @Override + public boolean isEditProjectAuth(Long projectId, String username) throws DSSProjectErrorException { + //校验当前登录用户是否含有修改权限 + QueryWrapper queryWrapper = new QueryWrapper(); + queryWrapper.eq("project_id",projectId); + queryWrapper.eq("username",username); + queryWrapper.ge("priv",ProjectUserPrivEnum.PRIV_EDIT.getRank());//编辑权限 + long count = projectUserMapper.selectCount(queryWrapper); + if(count == 0){ + DSSExceptionUtils.dealErrorException(ProjectServerResponse.PROJECT_NOT_EDIT_AUTH.getCode(),ProjectServerResponse.PROJECT_NOT_EDIT_AUTH.getMsg(), DSSProjectErrorException.class); + } + return true; + } + /** + * 根据用户名和工程id获取工程权限 + * @param projectId + * @param username + * @return + */ + @Override + public List getEditProjectList(Long projectId, String username){ + QueryWrapper queryWrapper = new QueryWrapper(); + queryWrapper.eq("project_id",projectId); + queryWrapper.eq("username",username); + return projectUserMapper.selectList(queryWrapper); + } + + /** + * 保存工程与用户关系 + * @param projectID + * @param request + */ + @Override + public void saveProjectUser(Long projectID,String username, ProjectCreateRequest request)throws Exception{ + //將创建人默认为发布权限 + List releaseUsers = request.getReleaseUsers(); + if(!request.getEditUsers().contains(username)){ + request.getEditUsers().add(username); + } + if(releaseUsers == null){ + releaseUsers = new ArrayList<>(); + } + if(!releaseUsers.contains(username)){ + releaseUsers.add(username); + } + + //批量保存 + saveBatch(request.getWorkspaceId(),projectID,releaseUsers,request.getEditUsers(),request.getAccessUsers()); + + //获取所有编辑权限的用户 + List sumEditUsers = ProjectUserUtils.getEditUserList(releaseUsers, request.getEditUsers()); + //去bml建一个工程 + bmlService.createBmlProject(username, request.getName(), sumEditUsers, request.getAccessUsers()); + } + + /** + * 修改工程与用户关系 + * @param dbProject + * @param modifyRequest + */ + @Override + public void modifyProjectUser(DSSProjectDO dbProject, ProjectModifyRequest modifyRequest, String loginuser) throws Exception{ + projectUserMapper.deleteAllPriv(dbProject.getId()); + //將创建人默认为发布权限 + List releaseUsers = modifyRequest.getReleaseUsers(); + String username = dbProject.getCreateBy(); + if(!modifyRequest.getEditUsers().contains(username)){ + modifyRequest.getEditUsers().add(username); + } + if(releaseUsers == null){ + releaseUsers = new ArrayList<>(); + } + if(!releaseUsers.contains(username)){ + releaseUsers.add(username); + } + + //批量保存 + saveBatch(modifyRequest.getWorkspaceId(),dbProject.getId(),releaseUsers,modifyRequest.getEditUsers(),modifyRequest.getAccessUsers()); + + //获取所有编辑权限的用户 + List sumEditUsers = ProjectUserUtils.getEditUserList(releaseUsers, modifyRequest.getEditUsers()); + //更新底层的权限 + bmlService.updateProjectPriv(dbProject.getName(), username, sumEditUsers, modifyRequest.getAccessUsers()); + } + + + + @Override + public List getListByParam(Long workspaceId, String username) { + QueryWrapper queryWrapper = new QueryWrapper<>(); + queryWrapper.eq("workspace_id",workspaceId); + queryWrapper.eq("username",username); + queryWrapper.ge("priv",ProjectUserPrivEnum.PRIV_ACCESS.getRank()); + List DSSProjectUserList = projectUserMapper.selectList(queryWrapper); + return DSSProjectUserList; + } + + //批量保存 + public void saveBatch(Long workspaceId,Long projectID,List releaseUsers,List editUsers,List accessUsers){ + List realReleaseUsers = releaseUsers.stream().distinct().collect(Collectors.toList()); + List realEditUsers = editUsers.stream().distinct().collect(Collectors.toList()); + List realAccessUsers = accessUsers.stream().distinct().collect(Collectors.toList()); + List addList = new ArrayList<>(); + addList.addAll(ProjectUserUtils.createPUser(workspaceId,projectID,realReleaseUsers,ProjectUserPrivEnum.PRIV_RELEASE.getRank())); + addList.addAll(ProjectUserUtils.createPUser(workspaceId,projectID,realEditUsers, ProjectUserPrivEnum.PRIV_EDIT.getRank())); + addList.addAll(ProjectUserUtils.createPUser(workspaceId,projectID,realAccessUsers,ProjectUserPrivEnum.PRIV_ACCESS.getRank())); + if(CollectionUtils.isEmpty(addList)){ + return; + } + //分批插入 + List tempAddList = new ArrayList<>(); + for (int i = 0; i < addList.size(); i++) { + tempAddList.add(addList.get(i)); + if (i > 0 && i % 200 == 0) { + projectUserMapper.insertBatchProjectUser(tempAddList); + tempAddList.clear(); + } + } + if (tempAddList.size() > 0) { + projectUserMapper.insertBatchProjectUser(tempAddList); + } + } + + @Override + public boolean isAdminByUsername(Long workspaceId, String username) { + //管理员ID，这里写死 + int roleId = 1; + return projectUserMapper.isAdminByUsername(workspaceId, username, roleId).longValue() > 0; + } + +} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/ApplicationArea.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ApplicationArea.java similarity index 91% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/ApplicationArea.java rename to dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ApplicationArea.java index aab3600d8..a06799f60 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/ApplicationArea.java +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ApplicationArea.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,7 +14,7 @@ * */ -package com.webank.wedatasphere.dss.server.entity; +package com.webank.wedatasphere.dss.framework.project.utils; public enum ApplicationArea { diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/LambdaHelper.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/LambdaHelper.java new file mode 100644 index 000000000..234d1bdad --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/LambdaHelper.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.utils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.function.Consumer; + + + +public class LambdaHelper { + + private static final Logger LOGGER = LoggerFactory.getLogger(LambdaHelper.class); + + public static Consumer consumerWrapper(Consumer consumer){ + return t -> { + try{ + consumer.accept(t); + }catch(Exception e){ + LOGGER.error("Failed to do with {} ", t, e); + } + }; + } + + +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ProjectStringUtils.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ProjectStringUtils.java new file mode 100644 index 000000000..4bbe90273 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ProjectStringUtils.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.utils; + + +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.net.URLEncoder; +import java.util.*; +import java.util.stream.Collectors; + +public class ProjectStringUtils { + + + private static final Logger LOGGER = LoggerFactory.getLogger(ProjectStringUtils.class); + + private static final String REDIRECT_FORMAT = "%s?redirect=%s&dssurl=${dssurl}&cookies=${cookies}"; + + public static final String MODE_SPLIT = ","; + public static final String KEY_SPLIT = "-"; + + public static List convertList(String str){ + if(org.apache.commons.lang.StringUtils.isEmpty(str)){ + return new ArrayList<>(); + } + return Arrays.stream(str.split(MODE_SPLIT)).map(String::trim).filter((s)-> StringUtils.isNotBlank(s)).distinct().collect(Collectors.toList()); + } + + //拼接 前后使用英文逗号结尾 + public static String getModeStr(List strList){ + if(CollectionUtils.isEmpty(strList)){ + return null; + } + return strList.stream().map(String::trim).filter((s)-> StringUtils.isNotBlank(s)).distinct().collect(Collectors.joining(MODE_SPLIT,MODE_SPLIT,MODE_SPLIT)); + } + + private static String URLEndoder(String str){ + try { + return URLEncoder.encode(str,"utf-8"); + } catch (UnsupportedEncodingException e) { + LOGGER.warn("endoe failed:",e); + return str; + } + } + + public static String redirectUrlFormat(String redirectUrl,String url){ + return String.format(REDIRECT_FORMAT,redirectUrl,URLEndoder(url)); + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ProjectUserUtils.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ProjectUserUtils.java new file mode 100644 index 000000000..2fe2ac253 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/ProjectUserUtils.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.utils; + +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectUser; +import org.apache.commons.collections.CollectionUtils; + +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +public class ProjectUserUtils { + + + public static List createPUser(Long wid, Long pid, List users, int priv){ + List retList = new ArrayList<>(); + if(CollectionUtils.isEmpty(users)){ + return retList; + } + users.forEach(tname->{ + DSSProjectUser DSSProjectUser = new DSSProjectUser(wid, pid, tname,priv); + retList.add(DSSProjectUser); + }); + return retList; + } + + + public static List getEditUserList(List releaseUsers, List editUsers){ + List sumEditUsers = new ArrayList<>(); + if (!CollectionUtils.isEmpty(releaseUsers)) { + sumEditUsers.addAll(releaseUsers); + } + if (!CollectionUtils.isEmpty(editUsers)) { + sumEditUsers.addAll(editUsers); + } + return sumEditUsers.stream().distinct().collect(Collectors.toList()); + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/RestfulUtils.java b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/RestfulUtils.java new file mode 100644 index 000000000..ad0115397 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/java/com/webank/wedatasphere/dss/framework/project/utils/RestfulUtils.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.utils; + +import com.webank.wedatasphere.linkis.server.Message; +import org.apache.commons.math3.util.Pair; + +import javax.ws.rs.core.Response; +import java.util.Arrays; + + +public class RestfulUtils { + + public static Response dealError(String reason){ + Message message = Message.error(reason); + return Message.messageToResponse(message); + } + + public static Response dealOk(String msg){ + Message message = Message.ok(msg); + return Message.messageToResponse(message); + } + + @SafeVarargs + public static Response dealOk(String msg, Pair... data){ + Message message = Message.ok(msg); + Arrays.stream(data).forEach(p -> message.data(p.getKey(), p.getValue())); + return Message.messageToResponse(message); + } + + +} diff --git a/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/DSSProjectServerApplication.scala b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/DSSProjectServerApplication.scala new file mode 100644 index 000000000..3e801a066 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/DSSProjectServerApplication.scala @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.server +import com.webank.wedatasphere.dss.common.utils.DSSMainHelper +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} + + +object DSSProjectServerApplication extends Logging { + + val userName: String = System.getProperty("user.name") + val hostName: String = Utils.getComputerName + + def main(args: Array[String]): Unit = { + val serviceName = System.getProperty("serviceName")//ProjectConf.SERVICE_NAME.getValue + DSSMainHelper.formatPropertyFiles(serviceName) + val allArgs = args ++ DSSMainHelper.getExtraSpringOptions + System.setProperty("hostName", hostName) + System.setProperty("userName", userName) + info(s"Ready to start $serviceName with args: ${allArgs.toList}.") + println(s"Test Ready to start $serviceName with args: ${allArgs.toList}.") + DataWorkCloudApplication.main(allArgs) + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/rpc/ProjectReceiver.scala b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/rpc/ProjectReceiver.scala new file mode 100644 index 000000000..1d0d3e89f --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/rpc/ProjectReceiver.scala @@ -0,0 +1,80 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.server.rpc + +import java.util + +import com.webank.wedatasphere.dss.common.entity.project.DSSProject +import com.webank.wedatasphere.dss.common.protocol.project.{ProjectInfoRequest, ProjectRelationRequest, ProjectRelationResponse} +import com.webank.wedatasphere.dss.framework.project.entity.DSSProjectDO +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceUserService +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestProjectImportOrchestrator +import com.webank.wedatasphere.linkis.protocol.usercontrol.{RequestUserListFromWorkspace, RequestUserWorkspace, ResponseUserWorkspace, ResponseWorkspaceUserList} +import com.webank.wedatasphere.linkis.rpc.{Receiver, Sender} +import org.springframework.beans.BeanUtils +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ +import scala.concurrent.duration.Duration + + +@Component +class ProjectReceiver(projectService: DSSProjectService) extends Receiver { + + @Autowired + var dssWorkspaceUserService:DSSWorkspaceUserService = _ + + + override def receive(message: Any, sender: Sender): Unit = { + + } + + override def receiveAndReply(message: Any, sender: Sender): Any = { + message match { + case projectRelationRequest: ProjectRelationRequest => + val dssProjectId = projectRelationRequest.getDssProjectId + val dssLabels = projectRelationRequest.getDssLabels + val appConnName = projectRelationRequest.getAppconnName + val appConnProjectId = projectService.getAppConnProjectId(dssProjectId, appConnName, dssLabels) + new ProjectRelationResponse(dssProjectId, appConnName, dssLabels, appConnProjectId) + case requestUserWorkspace:RequestUserWorkspace => + val userWorkspaceIds: util.List[Integer] = dssWorkspaceUserService.getUserWorkspaceIds(requestUserWorkspace.getUserName) + new ResponseUserWorkspace(userWorkspaceIds) + + case requestUserListFromWorkspace: RequestUserListFromWorkspace=> + val userList = requestUserListFromWorkspace.getUserWorkspaceIds.flatMap(id => dssWorkspaceUserService.getAllWorkspaceUsers(id)).distinct + new ResponseWorkspaceUserList(userList) + + case projectInfoRequest: ProjectInfoRequest => + val dssProjectDO: DSSProjectDO = projectService.getProjectById(projectInfoRequest.getProjectId) + val DSSProject = new DSSProject() + BeanUtils.copyProperties(dssProjectDO, DSSProject) + DSSProject + + case projectImportOrchestrator: RequestProjectImportOrchestrator => + projectService.importOrchestrator(projectImportOrchestrator) + } + } + + + override def receiveAndReply(message: Any, duration: Duration, sender: Sender): Any = { + null + } +} diff --git a/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/rpc/ProjectReceiverChooser.scala b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/rpc/ProjectReceiverChooser.scala new file mode 100644 index 000000000..350722ce8 --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/rpc/ProjectReceiverChooser.scala @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.server.rpc + +import com.webank.wedatasphere.dss.common.protocol.project.ProjectRelationRequest +import com.webank.wedatasphere.dss.framework.project.service.DSSProjectService +import com.webank.wedatasphere.linkis.rpc.{RPCMessageEvent, Receiver, ReceiverChooser} +import javax.annotation.PostConstruct +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.stereotype.Component + + +@Component +class ProjectReceiverChooser extends ReceiverChooser { + + + @Autowired + var projectService: DSSProjectService = _ + + private var receiver: Option[ProjectReceiver] = _ + + @PostConstruct + def init(): Unit = receiver = Some(new ProjectReceiver(projectService)) + + override def chooseReceiver(event: RPCMessageEvent): Option[Receiver] = event.message match { + case _: ProjectRelationRequest => receiver + case _ => None + } +} + + + + diff --git a/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/service/BMLService.scala b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/service/BMLService.scala new file mode 100644 index 000000000..3a5d837da --- /dev/null +++ b/dss-framework/dss-framework-project-server/src/main/scala/com/webank/wedatasphere/dss/framework/project/server/service/BMLService.scala @@ -0,0 +1,183 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.project.server.service + +import java.io.{ByteArrayInputStream, InputStream} +import java.util +import java.util.UUID + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException +import com.webank.wedatasphere.dss.common.utils.IoUtils +import com.webank.wedatasphere.linkis.bml.client.{BmlClient, BmlClientFactory} +import com.webank.wedatasphere.linkis.bml.protocol.{BmlDownloadResponse, BmlUpdateResponse, BmlUploadResponse} +import com.webank.wedatasphere.linkis.common.utils.{JavaLog, Logging, Utils} +import org.apache.commons.io.IOUtils +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ + + +@Component("projectServerBMLService") +class BMLService extends Logging{ + + def upload(userName: String, content: String, fileName: String, projectName:String): util.Map[String, Object] = { + val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) + val client: BmlClient = createBMLClient(userName) + val resource: BmlUploadResponse = client.uploadShareResource(userName, projectName, fileName, inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def upload(userName: String, inputStream: InputStream, fileName: String, projectName:String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + val resource: BmlUploadResponse = client.uploadShareResource(userName, projectName, fileName, inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def update(userName: String, resourceId: String, inputStream: InputStream): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + val resource: BmlUpdateResponse = client.updateShareResource(userName, resourceId, "", inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def update(userName: String, resourceId: String, content: String): util.Map[String, Object] = { + val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) + val client: BmlClient = createBMLClient(userName) + val resource: BmlUpdateResponse = client.updateShareResource(userName, resourceId, UUID.randomUUID().toString+".json", inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def query(userName: String, resourceId: String, version: String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + var resource: BmlDownloadResponse = null + if (version == null) { + resource = client.downloadShareResource(userName, resourceId) + } else { + resource = client.downloadShareResource(userName, resourceId, version) + } + if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") + val map = new util.HashMap[String, Object] + map += "path" -> resource.fullFilePath + map += "string" -> inputstremToString(resource.inputStream) + } + + def download(userName: String, resourceId: String, version: String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + var resource: BmlDownloadResponse = null + if (version == null) { + resource = client.downloadShareResource(userName, resourceId) + } else { + resource = client.downloadShareResource(userName, resourceId, version) + } + if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") + val map = new util.HashMap[String, Object] + map += "path" -> resource.fullFilePath + map += "is" -> resource.inputStream + } + + def downloadToLocalPath(userName: String, resourceId: String, version: String, path: String): String = { + val result = download(userName,resourceId,version) + val is = result.get("is").asInstanceOf[InputStream] + val os = IoUtils.generateExportOutputStream(path) + Utils.tryFinally(IOUtils.copy(is,os)){ + IOUtils.closeQuietly(os) + IOUtils.closeQuietly(is) + } + path + } + + def downloadAndGetFlowJson(userName: String, resourceId: String, version: String, path: String): String = { + downloadToLocalPath(userName,resourceId,version,path) + //因为下载到指定目录后返回的resource的Stream为null,只能从文件重新读取。 + val is = IoUtils.generateInputInputStream(path) + Utils.tryFinally(inputstremToString(is))(IOUtils.closeQuietly(is)) + } + + def readLocalResourceFile(userName: String,readPath: String): InputStream ={ + IoUtils.generateInputInputStream(readPath) + } + + def readLocalFlowJsonFile(userName: String,readPath: String): String ={ + var inputStream:InputStream = null + var flowJson:String = null + Utils.tryFinally{ + inputStream = IoUtils.generateInputInputStream(readPath) + flowJson = inputstremToString(inputStream) + }{ + IOUtils.closeQuietly(inputStream) + } + flowJson + } + + private def inputstremToString(inputStream: InputStream): String = { + scala.io.Source.fromInputStream(inputStream).mkString + } + + private def createBMLClient(userName: String): BmlClient = { + if (userName == null) + BmlClientFactory.createBmlClient() + else + BmlClientFactory.createBmlClient(userName) + } + + def createBmlProject(username:String, projectName:String, editUsers:util.List[String], + accessUsers:util.List[String] ): Unit ={ + val client = createBMLClient(username) + val response = client.createBmlProject(username, projectName, accessUsers, editUsers) + if (response.isSuccess){ + logger.info(s"for user $username create bml project $projectName success") + }else{ + logger.error(s"for user $username create bml project $projectName failed") + } + } + + def attachResourceAndProject(username:String, projectName:String, resourceId:String):Unit = { + val client = createBMLClient(username) + val response = client.attachResourceAndProject(projectName, resourceId) + if (response.isSuccess){ + logger.info(s"attach $username and $projectName success") + }else{ + logger.error(s"attach $username and $projectName failed") + } + } + + + def updateProjectPriv(projectName:String, username:String, editUsers:util.List[String], + accessUsers:util.List[String]): Unit ={ + val client = createBMLClient(username) + val response = client.updateProjectPriv(username, projectName, editUsers, accessUsers) + if (response.isSuccess){ + logger.info(s"attach $username and $projectName success") + }else{ + logger.error(s"attach $username and $projectName failed") + } + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/pom.xml b/dss-framework/dss-framework-workspace-server/pom.xml new file mode 100644 index 000000000..5541740e3 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/pom.xml @@ -0,0 +1,133 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + + 4.0.0 + + dss-framework-workspace-server + + + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + org.apache.commons + commons-lang3 + ${commons.lang3.version} + provided + + + com.webank.wedatasphere.linkis + linkis-httpclient + ${linkis.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + + + + com.webank.wedatasphere.dss + dss-framework-common + ${dss.version} + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + + org.apache.commons + commons-math3 + provided + + + com.webank.wedatasphere.dss + dss-appconn-manager-client + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + + 8 + 8 + + + + + + src/main/java + + **/*.xml + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/assembly/distribution.xml b/dss-framework/dss-framework-workspace-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..5303b0f55 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/assembly/distribution.xml @@ -0,0 +1,320 @@ + + + + dss-framework-workspace-server + + zip + + true + dss-framework-workspace-server + + + + + + lib + true + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ${basedir}/src/main/resources + + * + + 0777 + conf + unix + + + ${basedir}/bin + + * + + 0777 + bin + unix + + + . + + */** + + logs + + + + + diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/annotation/DSSWorkspaceUM.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/annotation/DSSWorkspaceUM.java new file mode 100644 index 000000000..74e80506d --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/annotation/DSSWorkspaceUM.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.annotation; + +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + + +@Retention(RetentionPolicy.RUNTIME) +@Target({ElementType.METHOD}) +public @interface DSSWorkspaceUM { + String value() default ""; +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/annotation/WorkspaceIdChecker.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/annotation/WorkspaceIdChecker.java new file mode 100644 index 000000000..3de0c1ef9 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/annotation/WorkspaceIdChecker.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.annotation; + +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + + +@Retention(RetentionPolicy.RUNTIME) +@Target({ElementType.METHOD}) +public @interface WorkspaceIdChecker { + String value() default ""; +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSPrivCheckerAspect.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSPrivCheckerAspect.java new file mode 100644 index 000000000..ff8e4b818 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSPrivCheckerAspect.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.aop; + +import org.aspectj.lang.annotation.Aspect; +import org.springframework.stereotype.Component; + + +@Aspect +@Component +public class DSSPrivCheckerAspect { + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSUWorkspaceIdCheckerAspect.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSUWorkspaceIdCheckerAspect.java new file mode 100644 index 000000000..3f09b9f16 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSUWorkspaceIdCheckerAspect.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.aop; + +import org.aspectj.lang.annotation.Aspect; +import org.springframework.stereotype.Component; + + +@Aspect +@Component +public class DSSUWorkspaceIdCheckerAspect { + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSWorkspaceUMAspect.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSWorkspaceUMAspect.java new file mode 100644 index 000000000..e42e2b6a9 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/aop/DSSWorkspaceUMAspect.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.aop; + +import org.aspectj.lang.annotation.Aspect; +import org.springframework.stereotype.Component; + + +@Aspect +@Component +public class DSSWorkspaceUMAspect { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/BaseEntity.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/BaseEntity.java new file mode 100644 index 000000000..0f9960560 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/BaseEntity.java @@ -0,0 +1,63 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class BaseEntity { + + private String createBy; + + private Date createTime = new Date(); + + private Date lastUpdateTime = new Date(); + + private String lastUpdateUser; + + public String getCreateBy() { + return createBy; + } + + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(Date lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } + + public String getLastUpdateUser() { + return lastUpdateUser; + } + + public void setLastUpdateUser(String lastUpdateUser) { + this.lastUpdateUser = lastUpdateUser; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSApplication.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSApplication.java new file mode 100644 index 000000000..a7b214b2d --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSApplication.java @@ -0,0 +1,169 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; + + +@TableName(value = "dss_application") +public class DSSApplication implements Serializable { + + private static final long serialVersionUID=1L; + + @TableId(value = "id", type = IdType.AUTO) + private Integer id; + + private String name; + + private String url; + + private Boolean isUserNeedInit; + + private Integer level; + + private String userInitUrl; + + private Boolean existsProjectService; + + private String projectUrl; + + private String enhanceJson; + + private Boolean ifIframe; + + private String homepageUrl; + + private String redirectUrl; + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public Boolean getUserNeedInit() { + return isUserNeedInit; + } + + public void setUserNeedInit(Boolean userNeedInit) { + isUserNeedInit = userNeedInit; + } + + public Integer getLevel() { + return level; + } + + public void setLevel(Integer level) { + this.level = level; + } + + public String getUserInitUrl() { + return userInitUrl; + } + + public void setUserInitUrl(String userInitUrl) { + this.userInitUrl = userInitUrl; + } + + public Boolean getExistsProjectService() { + return existsProjectService; + } + + public void setExistsProjectService(Boolean existsProjectService) { + this.existsProjectService = existsProjectService; + } + + public String getProjectUrl() { + return projectUrl; + } + + public void setProjectUrl(String projectUrl) { + this.projectUrl = projectUrl; + } + + public String getEnhanceJson() { + return enhanceJson; + } + + public void setEnhanceJson(String enhanceJson) { + this.enhanceJson = enhanceJson; + } + + public Boolean getIfIframe() { + return ifIframe; + } + + public void setIfIframe(Boolean ifIframe) { + this.ifIframe = ifIframe; + } + + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + public String getRedirectUrl() { + return redirectUrl; + } + + public void setRedirectUrl(String redirectUrl) { + this.redirectUrl = redirectUrl; + } + + @Override + public String toString() { + return "DssApplication{" + + "id=" + id + + ", name='" + name + '\'' + + ", url='" + url + '\'' + + ", isUserNeedInit=" + isUserNeedInit + + ", level=" + level + + ", userInitUrl='" + userInitUrl + '\'' + + ", existsProjectService=" + existsProjectService + + ", projectUrl='" + projectUrl + '\'' + + ", enhanceJson='" + enhanceJson + '\'' + + ", ifIframe=" + ifIframe + + ", homepageUrl='" + homepageUrl + '\'' + + ", redirectUrl='" + redirectUrl + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSApplicationBean.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSApplicationBean.java new file mode 100644 index 000000000..c0ead4431 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSApplicationBean.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + + +public class DSSApplicationBean { + private Integer id; + private String name; + private String homepageUrl; + private String projectUrl; + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + public String getProjectUrl() { + return projectUrl; + } + + public void setProjectUrl(String projectUrl) { + this.projectUrl = projectUrl; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSComponentRole.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSComponentRole.java new file mode 100644 index 000000000..9cac8a268 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSComponentRole.java @@ -0,0 +1,102 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + +@TableName(value = "dss_component_role") +public class DSSComponentRole implements Serializable { + private static final long serialVersionUID=1L; + + @TableId(value = "id", type = IdType.AUTO) + private Long id; + + private Long workspaceId; + + private Integer componentId; + + private Integer roleId; + + private Integer priv; + + private Date updateTime; + + private String updateby; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Integer getComponentId() { + return componentId; + } + + public void setComponentId(Integer componentId) { + this.componentId = componentId; + } + + public Integer getRoleId() { + return roleId; + } + + public void setRoleId(Integer roleId) { + this.roleId = roleId; + } + + public Integer getPriv() { + return priv; + } + + public void setPriv(Integer priv) { + this.priv = priv; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdateby() { + return updateby; + } + + public void setUpdateby(String updateby) { + this.updateby = updateby; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSDictionary.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSDictionary.java new file mode 100644 index 000000000..6214e75b4 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSDictionary.java @@ -0,0 +1,304 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + +@TableName(value = "dss_dictionary") +public class DSSDictionary implements Serializable { + + private static final long serialVersionUID=1L; + + /** + * 主键ID，默认为0，所有空间都有 + */ + @TableId(value = "id", type = IdType.AUTO) + private Integer id; + + /** + * 空间ID，默认为0，所有空间都有 + */ + private Integer workspaceId; + + /** + * 父key + */ + private String parentKey; + + /** + * 名称 + */ + private String dicName; + /** + * 名称(英文) + */ + private String dicNameEn; + + /** + * key 相当于编码，空间是w_开头，工程是p_ + */ + private String dicKey; + + /** + * key对应的值 + */ + private String dicValue; + + /** + * key对应的值(英文) + */ + private String dicValueEn; + + /** + * 标题 + */ + private String title; + + /** + * 标题(英文) + */ + private String titleEn; + + /** + * url + */ + private String url; + + /** + * url类型: 0-内部系统，1-外部系统；默认是内部 + */ + private Integer urlType; + + /** + * 图标 + */ + private String icon; + + /** + * 序号 + */ + private Integer orderNum; + + /** + * 备注 + */ + private String remark; + + /** + * 创建人 + */ + private String createUser; + + /** + * 创建时间 + */ + private Date createTime; + + /** + * 更新人 + */ + private String updateUser; + + /** + * 更新时间 + */ + private Date updateTime; + + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public String getParentKey() { + return parentKey; + } + + public void setParentKey(String parentKey) { + this.parentKey = parentKey; + } + + public String getDicName() { + return dicName; + } + + public void setDicName(String dicName) { + this.dicName = dicName; + } + + public String getDicKey() { + return dicKey; + } + + public void setDicKey(String dicKey) { + this.dicKey = dicKey; + } + + public String getDicValue() { + return dicValue; + } + + public void setDicValue(String dicValue) { + this.dicValue = dicValue; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public Integer getUrlType() { + return urlType; + } + + public void setUrlType(Integer urlType) { + this.urlType = urlType; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public Integer getOrderNum() { + return orderNum; + } + + public void setOrderNum(Integer orderNum) { + this.orderNum = orderNum; + } + + public String getRemark() { + return remark; + } + + public void setRemark(String remark) { + this.remark = remark; + } + + public String getCreateUser() { + return createUser; + } + + public void setCreateUser(String createUser) { + this.createUser = createUser; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getUpdateUser() { + return updateUser; + } + + public void setUpdateUser(String updateUser) { + this.updateUser = updateUser; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getDicNameEn() { + return dicNameEn; + } + + public void setDicNameEn(String dicNameEn) { + this.dicNameEn = dicNameEn; + } + + public String getDicValueEn() { + return dicValueEn; + } + + public void setDicValueEn(String dicValueEn) { + this.dicValueEn = dicValueEn; + } + + public String getTitleEn() { + return titleEn; + } + + public void setTitleEn(String titleEn) { + this.titleEn = titleEn; + } + + @Override + public String toString() { + return "DSSDictionary{" + + "id=" + id + + ", workspaceId=" + workspaceId + + ", parentKey='" + parentKey + '\'' + + ", dicName='" + dicName + '\'' + + ", dicNameEn='" + dicNameEn + '\'' + + ", dicKey='" + dicKey + '\'' + + ", dicValue='" + dicValue + '\'' + + ", dicValueEn='" + dicValueEn + '\'' + + ", title='" + title + '\'' + + ", titleEn='" + titleEn + '\'' + + ", url='" + url + '\'' + + ", urlType=" + urlType + + ", icon='" + icon + '\'' + + ", orderNum=" + orderNum + + ", remark='" + remark + '\'' + + ", createUser='" + createUser + '\'' + + ", createTime=" + createTime + + ", updateUser='" + updateUser + '\'' + + ", updateTime=" + updateTime + + '}'; + } +} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSFavorite.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSFavorite.java similarity index 89% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSFavorite.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSFavorite.java index 9d9956442..40f691d8c 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSFavorite.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSFavorite.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -14,11 +13,9 @@ * limitations under the License. * */ -package com.webank.wedatasphere.dss.server.entity; +package com.webank.wedatasphere.dss.framework.workspace.bean; + -/** - * Created by Adamyuanyuan on 2020/6/25 - */ public class DSSFavorite extends BaseEntity{ private Long id; diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSHomepage.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSHomepage.java new file mode 100644 index 000000000..48ab10de3 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSHomepage.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + + +public class DSSHomepage { + private int id; + private String name; + private String url; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSMenu.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSMenu.java new file mode 100644 index 000000000..d3c1ba28f --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSMenu.java @@ -0,0 +1,169 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableField; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; +import com.google.common.base.Objects; + +import java.io.Serializable; + + +@TableName(value = "dss_menu") +public class DSSMenu implements Serializable { + private static final long serialVersionUID=1L; + @TableId(value = "id", type = IdType.AUTO) + private Integer id; + private String name; + private int level; + private int upperMenuId; + private String frontName; + private String comment; + private String description; + private boolean isActive; + private boolean isComponent; + @TableField(exist = false) + private String url; + private String icon; + private int applicationId; + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public int getLevel() { + return level; + } + + public void setLevel(int level) { + this.level = level; + } + + public int getUpperMenuId() { + return upperMenuId; + } + + public void setUpperMenuId(int upperMenuId) { + this.upperMenuId = upperMenuId; + } + + public String getFrontName() { + return frontName; + } + + public void setFrontName(String frontName) { + this.frontName = frontName; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public boolean isActive() { + return isActive; + } + + public void setActive(boolean active) { + isActive = active; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public boolean isComponent() { + return isComponent; + } + + public void setComponent(boolean component) { + isComponent = component; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public int getApplicationId() { + return applicationId; + } + + public void setApplicationId(int applicationId) { + this.applicationId = applicationId; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + + if (o == null || getClass() != o.getClass()) { + return false; + } + DSSMenu dssMenu = (DSSMenu) o; + return id == dssMenu.id && + level == dssMenu.level && + upperMenuId == dssMenu.upperMenuId && + isActive == dssMenu.isActive && + isComponent == dssMenu.isComponent && + applicationId == dssMenu.applicationId && + Objects.equal(name, dssMenu.name) && + Objects.equal(frontName, dssMenu.frontName) && + Objects.equal(description, dssMenu.description) && + Objects.equal(url, dssMenu.url) && + Objects.equal(icon, dssMenu.icon); + } + + @Override + public int hashCode() { + return Objects.hashCode(id, name, level, upperMenuId, frontName, description, isActive, isComponent, url, icon, applicationId); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSMenuRole.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSMenuRole.java new file mode 100644 index 000000000..69dca39de --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSMenuRole.java @@ -0,0 +1,115 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + +@TableName(value = "dss_menu_role") +public class DSSMenuRole implements Serializable { + + private static final long serialVersionUID=1L; + + @TableId(value = "id", type = IdType.AUTO) + private Integer id; + + private Integer workspaceId; + + private Integer menuId; + + private Integer roleId; + + private Integer priv; + + private Date updateTime; + + private String updateby; + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public Integer getMenuId() { + return menuId; + } + + public void setMenuId(Integer menuId) { + this.menuId = menuId; + } + + public Integer getRoleId() { + return roleId; + } + + public void setRoleId(Integer roleId) { + this.roleId = roleId; + } + + public Integer getPriv() { + return priv; + } + + public void setPriv(Integer priv) { + this.priv = priv; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdateby() { + return updateby; + } + + public void setUpdateby(String updateby) { + this.updateby = updateby; + } + + @Override + public String toString() { + return "DssMenuRole{" + + "id=" + id + + ", workspaceId=" + workspaceId + + ", menuId=" + menuId + + ", roleId=" + roleId + + ", priv=" + priv + + ", updateTime=" + updateTime + + ", updateby='" + updateby + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSRole.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSRole.java new file mode 100644 index 000000000..152fe0573 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSRole.java @@ -0,0 +1,89 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSRole { + private int id; + private int workspaceId; + private String name; + private String frontName; + private Date createTime; + private String description; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getFrontName() { + return frontName; + } + + public void setFrontName(String frontName) { + this.frontName = frontName; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + @Override + public String toString() { + return "DSSRole{" + + "id=" + id + + ", workspaceId=" + workspaceId + + ", name='" + name + '\'' + + ", frontName='" + frontName + '\'' + + ", createTime=" + createTime + + ", description='" + description + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSUser.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSUser.java new file mode 100644 index 000000000..4820b7c29 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSUser.java @@ -0,0 +1,93 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableField; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; + + +@TableName(value = "dss_user") +public class DSSUser implements Serializable { + + private static final long serialVersionUID = -5404077122214161197L; + + @TableId(type = IdType.AUTO) + private Long id; + + private String username; + + private String name; + + private Boolean isFirstLogin; + + private Boolean isAdmin; + + public DSSUser() { + } + + public DSSUser(Long id, String username, String name, Boolean isFirstLogin, Boolean isAdmin) { + this.id = id; + this.username = username; + this.name = name; + this.isFirstLogin = isFirstLogin; + this.isAdmin = isAdmin; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Boolean getFirstLogin() { + return isFirstLogin; + } + + public void setFirstLogin(Boolean firstLogin) { + isFirstLogin = firstLogin; + } + + public Boolean getAdmin() { + return isAdmin; + } + + public void setAdmin(Boolean admin) { + isAdmin = admin; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspace.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspace.java new file mode 100644 index 000000000..d0731b7b6 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspace.java @@ -0,0 +1,124 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspace { + private int id; + private String name; + private String label; + private String description; + private String createBy; + private Date createTime; + private String department; + private String product; + private String source; + private Date lastUpdateTime; + private String lastUpdateUser; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getLabel() { + return label; + } + + public void setLabel(String label) { + this.label = label; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getCreateBy() { + return createBy; + } + + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getDepartment() { + return department; + } + + public void setDepartment(String department) { + this.department = department; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public Date getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(Date lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } + + public String getLastUpdateUser() { + return lastUpdateUser; + } + + public void setLastUpdateUser(String lastUpdateUser) { + this.lastUpdateUser = lastUpdateUser; + } + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponent.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponent.java new file mode 100644 index 000000000..39e791a45 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponent.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + + +public class DSSWorkspaceComponent { + private int id; + private String componentName; + private String homepageUrl; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getComponentName() { + return componentName; + } + + public void setComponentName(String componentName) { + this.componentName = componentName; + } + + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentInfo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentInfo.java new file mode 100644 index 000000000..d7040c83b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentInfo.java @@ -0,0 +1,106 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + + +import java.util.List; + + +public class DSSWorkspaceComponentInfo { + private String title; + + private String icon; + + private String desc; + + private String buttonText; +/* + private List statistics;*/ + + private String componentUrl; + + private String userManualUrl; + + private String indicatorUrl; + + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public String getDesc() { + return desc; + } + + public void setDesc(String desc) { + this.desc = desc; + } + + public String getButtonText() { + return buttonText; + } + + public void setButtonText(String buttonText) { + this.buttonText = buttonText; + } + +/* + public List getStatistics() { + return statistics; + } + + public void setStatistics(List statistics) { + this.statistics = statistics; + } +*/ + + public String getComponentUrl() { + return componentUrl; + } + + public void setComponentUrl(String componentUrl) { + this.componentUrl = componentUrl; + } + + public String getUserManualUrl() { + return userManualUrl; + } + + public void setUserManualUrl(String userManualUrl) { + this.userManualUrl = userManualUrl; + } + + public String getIndicatorUrl() { + return indicatorUrl; + } + + public void setIndicatorUrl(String indicatorUrl) { + this.indicatorUrl = indicatorUrl; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentPriv.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentPriv.java new file mode 100644 index 000000000..3b05f0037 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentPriv.java @@ -0,0 +1,100 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspaceComponentPriv { + + private int id; + private int workspaceId; + private int roleId; + private int componentId; + private int priv; + private Date updateTime; + private String updateBy; + + + public DSSWorkspaceComponentPriv() { + } + + public DSSWorkspaceComponentPriv(int workspaceId, int roleId, int componentId, int priv, Date updateTime, String updateBy) { + this.workspaceId = workspaceId; + this.roleId = roleId; + this.componentId = componentId; + this.priv = priv; + this.updateTime = updateTime; + this.updateBy = updateBy; + } + + public int getComponentId() { + return componentId; + } + + public void setComponentId(int componentId) { + this.componentId = componentId; + } + + public int getPriv() { + return priv; + } + + public void setPriv(int priv) { + this.priv = priv; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdateBy() { + return updateBy; + } + + public void setUpdateBy(String updateBy) { + this.updateBy = updateBy; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentRolePriv.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentRolePriv.java new file mode 100644 index 000000000..87fdd4a7e --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceComponentRolePriv.java @@ -0,0 +1,106 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.google.common.base.Objects; + +import java.util.Date; + + +public class DSSWorkspaceComponentRolePriv { + private Integer id; + private Integer workspaceId; + private Integer componentId; + private Integer roleId; + private Integer priv; + private Date lastUpdateTime; + private String updateBy; + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public Integer getComponentId() { + return componentId; + } + + public void setComponentId(Integer componentId) { + this.componentId = componentId; + } + + public Integer getRoleId() { + return roleId; + } + + public void setRoleId(Integer roleId) { + this.roleId = roleId; + } + + public Integer getPriv() { + return priv; + } + + public void setPriv(Integer priv) { + this.priv = priv; + } + + public Date getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(Date lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } + + public String getUpdateBy() { + return updateBy; + } + + public void setUpdateBy(String updateBy) { + this.updateBy = updateBy; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + DSSWorkspaceComponentRolePriv that = (DSSWorkspaceComponentRolePriv) o; + return Objects.equal(componentId, that.componentId) && + Objects.equal(roleId, that.roleId); + } + + @Override + public int hashCode() { + return Objects.hashCode(componentId, roleId); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceHomepage.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceHomepage.java new file mode 100644 index 000000000..22e218cd5 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceHomepage.java @@ -0,0 +1,88 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.time.LocalDateTime; +import java.util.Date; + + +@TableName(value = "dss_workspace_homepage") +public class DSSWorkspaceHomepage { + private static final long serialVersionUID=1L; + @TableId(value = "id", type = IdType.AUTO) + private Integer id; + private Integer workspaceId; + private Integer roleId; + private String homepageUrl; + private Date updateTime; + + + public DSSWorkspaceHomepage(){ + + } + + public DSSWorkspaceHomepage(Integer workspaceId, Integer roleId, String homepageUrl, Date updateTime) { + this.workspaceId = workspaceId; + this.roleId = roleId; + this.homepageUrl = homepageUrl; + this.updateTime = updateTime; + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public Integer getRoleId() { + return roleId; + } + + public void setRoleId(Integer roleId) { + this.roleId = roleId; + } + + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceHomepageSetting.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceHomepageSetting.java new file mode 100644 index 000000000..899741f72 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceHomepageSetting.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspaceHomepageSetting { + private Integer id; + private Integer workspaceId; + private Integer roleId; + private String homepageUrl; + private Date updateTime; + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public Integer getRoleId() { + return roleId; + } + + public void setRoleId(Integer roleId) { + this.roleId = roleId; + } + + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuComponentUrl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuComponentUrl.java new file mode 100644 index 000000000..23348c5bf --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuComponentUrl.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspaceMenuComponentUrl { + private int id; + private int menuId; + private int dssApplicationId; + private String url; + private String manulUrl; + private String operationUrl; + private Date updateTime; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getMenuId() { + return menuId; + } + + public void setMenuId(int menuId) { + this.menuId = menuId; + } + + public int getDssApplicationId() { + return dssApplicationId; + } + + public void setDssApplicationId(int dssApplicationId) { + this.dssApplicationId = dssApplicationId; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public String getManulUrl() { + return manulUrl; + } + + public void setManulUrl(String manulUrl) { + this.manulUrl = manulUrl; + } + + public String getOperationUrl() { + return operationUrl; + } + + public void setOperationUrl(String operationUrl) { + this.operationUrl = operationUrl; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuRole.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuRole.java new file mode 100644 index 000000000..74c51704f --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuRole.java @@ -0,0 +1,95 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspaceMenuRole { + private int id; + private int workspaceId; + private int menuId; + private int roleId; + private int priv; + private Date updateTime; + private String updateBy; + + public DSSWorkspaceMenuRole(int workspaceId, int menuId, int roleId, int priv, Date updateTime, String updateBy) { + this.workspaceId = workspaceId; + this.menuId = menuId; + this.roleId = roleId; + this.priv = priv; + this.updateTime = updateTime; + this.updateBy = updateBy; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public int getMenuId() { + return menuId; + } + + public void setMenuId(int menuId) { + this.menuId = menuId; + } + + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public int getPriv() { + return priv; + } + + public void setPriv(int priv) { + this.priv = priv; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdateBy() { + return updateBy; + } + + public void setUpdateBy(String updateBy) { + this.updateBy = updateBy; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuRolePriv.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuRolePriv.java new file mode 100644 index 000000000..f380e5449 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceMenuRolePriv.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspaceMenuRolePriv { + private int id; + private int workspaceId; + private int menuId; + private int roleId; + private int priv; + private Date updateTime; + private String updateBy; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public int getMenuId() { + return menuId; + } + + public void setMenuId(int menuId) { + this.menuId = menuId; + } + + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public int getPriv() { + return priv; + } + + public void setPriv(int priv) { + this.priv = priv; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdateBy() { + return updateBy; + } + + public void setUpdateBy(String updateBy) { + this.updateBy = updateBy; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceUser.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceUser.java new file mode 100644 index 000000000..8f5756908 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/DSSWorkspaceUser.java @@ -0,0 +1,63 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class DSSWorkspaceUser { + + private String username; + + private int workspaceId; + + private String creator; + + private Date joinTime; + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Date getJoinTime() { + return joinTime; + } + + public void setJoinTime(Date joinTime) { + this.joinTime = joinTime; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/LinkisUser.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/LinkisUser.java new file mode 100644 index 000000000..ec6aa9c9b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/LinkisUser.java @@ -0,0 +1,84 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.util.Date; + +@TableName(value = "linkis_user") +public class LinkisUser { + @TableId(type = IdType.AUTO) + private Long id; + private String username; + private String name; + private Boolean isFirstLogin; + private Date updateTime; + private Date createTime; + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Boolean getFirstLogin() { + return isFirstLogin; + } + + public void setFirstLogin(Boolean firstLogin) { + isFirstLogin = firstLogin; + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/Sidebar.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/Sidebar.java new file mode 100644 index 000000000..8a743fdd6 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/Sidebar.java @@ -0,0 +1,185 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + +@TableName(value = "dss_sidebar") +public class Sidebar implements Serializable { + + private static final long serialVersionUID = -5404077122214161197L; + + @TableId(type = IdType.AUTO) + private Integer id; + + //名称 + private String name; + + //名称(英文) + private String nameEn; + + //标题 + private String title; + + //标题(英文) + private String titleEn; + + //类型: 0-知识库，1-菜单，2-常见问题 + private Integer type; + + //序号 + private Integer orderNum; + + //备注 + private String remark; + + //创建人 + private String createUser; + + //创建时间 + private Date createTime; + + //更新人 + private String updateUser; + + //更新时间 + private Date updateTime; + + + public Sidebar() { + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public Integer getType() { + return type; + } + + public void setType(Integer type) { + this.type = type; + } + + public Integer getOrderNum() { + return orderNum; + } + + public void setOrderNum(Integer orderNum) { + this.orderNum = orderNum; + } + + public String getRemark() { + return remark; + } + + public void setRemark(String remark) { + this.remark = remark; + } + + public String getCreateUser() { + return createUser; + } + + public void setCreateUser(String createUser) { + this.createUser = createUser; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getUpdateUser() { + return updateUser; + } + + public void setUpdateUser(String updateUser) { + this.updateUser = updateUser; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getNameEn() { + return nameEn; + } + + public void setNameEn(String nameEn) { + this.nameEn = nameEn; + } + + public String getTitleEn() { + return titleEn; + } + + public void setTitleEn(String titleEn) { + this.titleEn = titleEn; + } + + @Override + public String toString() { + return "Sidebar{" + + "id=" + id + + ", name='" + name + '\'' + + ", nameEn='" + nameEn + '\'' + + ", title='" + title + '\'' + + ", titleEn='" + titleEn + '\'' + + ", type=" + type + + ", orderNum=" + orderNum + + ", remark='" + remark + '\'' + + ", createUser='" + createUser + '\'' + + ", createTime=" + createTime + + ", updateUser='" + updateUser + '\'' + + ", updateTime=" + updateTime + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/SidebarContent.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/SidebarContent.java new file mode 100644 index 000000000..100ba03bb --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/SidebarContent.java @@ -0,0 +1,223 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import com.baomidou.mybatisplus.annotation.IdType; +import com.baomidou.mybatisplus.annotation.TableId; +import com.baomidou.mybatisplus.annotation.TableName; + +import java.io.Serializable; +import java.util.Date; + + +@TableName(value = "dss_sidebar_content") +public class SidebarContent implements Serializable { + + private static final long serialVersionUID = -5404077122214161197L; + + @TableId(type = IdType.AUTO) + private Integer id; + + //侧边栏ID + private Integer sidebarId; + + //名称 + private String name; + + //名称(英文) + private String nameEn; + + //标题 + private String title; + + //标题(英文) + private String titleEn; + + //url + private String url; + + /** + * url类型: 0-内部系统，1-外部系统；默认是内部 + */ + private Integer urlType; + + //icon + private String icon; + + //序号 + private Integer orderNum; + + //备注 + private String remark; + + //创建人 + private String createUser; + + //创建时间 + private Date createTime; + + //更新人 + private String updateUser; + + //更新时间 + private Date updateTime; + + + public SidebarContent() { + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getSidebarId() { + return sidebarId; + } + + public void setSidebarId(Integer sidebarId) { + this.sidebarId = sidebarId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public Integer getUrlType() { + return urlType; + } + + public void setUrlType(Integer urlType) { + this.urlType = urlType; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public Integer getOrderNum() { + return orderNum; + } + + public void setOrderNum(Integer orderNum) { + this.orderNum = orderNum; + } + + public String getRemark() { + return remark; + } + + public void setRemark(String remark) { + this.remark = remark; + } + + public String getCreateUser() { + return createUser; + } + + public void setCreateUser(String createUser) { + this.createUser = createUser; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getUpdateUser() { + return updateUser; + } + + public void setUpdateUser(String updateUser) { + this.updateUser = updateUser; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getNameEn() { + return nameEn; + } + + public void setNameEn(String nameEn) { + this.nameEn = nameEn; + } + + public String getTitleEn() { + return titleEn; + } + + public void setTitleEn(String titleEn) { + this.titleEn = titleEn; + } + + @Override + public String toString() { + return "SidebarContent{" + + "id=" + id + + ", sidebarId=" + sidebarId + + ", name='" + name + '\'' + + ", nameEn='" + nameEn + '\'' + + ", title='" + title + '\'' + + ", titleEn='" + titleEn + '\'' + + ", url='" + url + '\'' + + ", urlType=" + urlType + + ", icon='" + icon + '\'' + + ", orderNum=" + orderNum + + ", remark='" + remark + '\'' + + ", createUser='" + createUser + '\'' + + ", createTime=" + createTime + + ", updateUser='" + updateUser + '\'' + + ", updateTime=" + updateTime + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/StaffInfo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/StaffInfo.java new file mode 100644 index 000000000..7200c12da --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/StaffInfo.java @@ -0,0 +1,147 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import org.codehaus.jackson.annotate.JsonIgnoreProperties; +import org.codehaus.jackson.annotate.JsonProperty; + + +@JsonIgnoreProperties(ignoreUnknown = true) +public class StaffInfo { + @JsonProperty(value = "ID") + private String id; + @JsonProperty(value = "StaffID") + private String staffId; + @JsonProperty(value = "OID") + private String oId; + @JsonProperty(value = "ChineseName") + private String chineseName; + @JsonProperty(value = "EnglishName") + private String englishName; + @JsonProperty(value = "FullName") + private String fullName; + @JsonProperty(value = "OrgName") + private String orgName; + @JsonProperty(value = "OrgFullName") + private String orgFullName; + @JsonProperty(value = "Status") + private String status; + @JsonProperty(value = "PersonGroup") + private String personGroup; + + public StaffInfo(String id,String englishName, String orgFullName){ + this.id = id; + this.englishName = englishName; + this.orgFullName = orgFullName; + } + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getStaffId() { + return staffId; + } + + public void setStaffId(String staffId) { + this.staffId = staffId; + } + + public String getoId() { + return oId; + } + + public void setoId(String oId) { + this.oId = oId; + } + + public String getChineseName() { + return chineseName; + } + + public void setChineseName(String chineseName) { + this.chineseName = chineseName; + } + + public String getEnglishName() { + return englishName; + } + + public void setEnglishName(String englishName) { + this.englishName = englishName; + } + + public String getFullName() { + return fullName; + } + + public void setFullName(String fullName) { + this.fullName = fullName; + } + + public String getOrgName() { + return orgName; + } + + public void setOrgName(String orgName) { + this.orgName = orgName; + } + + public String getOrgFullName() { + return orgFullName; + } + + public void setOrgFullName(String orgFullName) { + this.orgFullName = orgFullName; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public String getPersonGroup() { + return personGroup; + } + + public void setPersonGroup(String personGroup) { + this.personGroup = personGroup; + } + + @Override + public String toString() { + return "StaffInfo{" + + "id='" + id + '\'' + + ", staffId='" + staffId + '\'' + + ", oId='" + oId + '\'' + + ", chineseName='" + chineseName + '\'' + + ", englishName='" + englishName + '\'' + + ", fullName='" + fullName + '\'' + + ", orgName='" + orgName + '\'' + + ", orgFullName='" + orgFullName + '\'' + + ", status='" + status + '\'' + + ", personGroup='" + personGroup + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/WorkSpaceUserAndRole.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/WorkSpaceUserAndRole.java new file mode 100644 index 000000000..d2fef62de --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/WorkSpaceUserAndRole.java @@ -0,0 +1,80 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean; + +import java.util.Date; + + +public class WorkSpaceUserAndRole { + + private int id; + private int roleId; + private int workspaceId; + private Long userID; + private String creator; + private Date date; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getUserID() { + return userID; + } + + public void setUserID(Long userID) { + this.userID = userID; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Date getDate() { + return date; + } + + public void setDate(Date date) { + this.date = date; + } + + +} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageDemoInstanceVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageDemoInstanceVo.java similarity index 91% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageDemoInstanceVo.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageDemoInstanceVo.java index b880f0f2a..2845aba13 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageDemoInstanceVo.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageDemoInstanceVo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -14,11 +13,9 @@ * limitations under the License. * */ -package com.webank.wedatasphere.dss.server.dto.response; +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; + -/** - * Created by schumiyi on 2020/6/23 - */ public class HomepageDemoInstanceVo { private Long id; diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageDemoMenuVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageDemoMenuVo.java similarity index 91% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageDemoMenuVo.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageDemoMenuVo.java index c7d4fdcbd..04b7558c5 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageDemoMenuVo.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageDemoMenuVo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -14,13 +13,11 @@ * limitations under the License. * */ -package com.webank.wedatasphere.dss.server.dto.response; +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; import java.util.List; -/** - * Created by schumiyi on 2020/6/23 - */ + public class HomepageDemoMenuVo { private Long id; diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageVideoVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageVideoVo.java similarity index 89% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageVideoVo.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageVideoVo.java index 0f9ce857d..956c039f4 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/HomepageVideoVo.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/HomepageVideoVo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -14,11 +13,9 @@ * limitations under the License. * */ -package com.webank.wedatasphere.dss.server.dto.response; +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; + -/** - * Created by schumiyi on 2020/6/23 - */ public class HomepageVideoVo { private Long id; diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/OnestopMenuAppInstanceVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/OnestopMenuAppInstanceVo.java similarity index 83% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/OnestopMenuAppInstanceVo.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/OnestopMenuAppInstanceVo.java index f22ca6b33..6f093bf02 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/OnestopMenuAppInstanceVo.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/OnestopMenuAppInstanceVo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -14,11 +13,11 @@ * limitations under the License. * */ -package com.webank.wedatasphere.dss.server.dto.response; +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; + +import java.util.Map; + -/** - * Created by schumiyi on 2020/6/24 - */ public class OnestopMenuAppInstanceVo { private Long id; private String title; @@ -33,6 +32,17 @@ public class OnestopMenuAppInstanceVo { private Boolean isActive; private String icon; private Integer order; + private Map nameAndUrls; + //图片 + private String image; + + public String getImage() { + return image; + } + + public void setImage(String image) { + this.image = image; + } public Long getId() { return id; @@ -137,4 +147,12 @@ public String getName() { public void setName(String name) { this.name = name; } + + public Map getNameAndUrls() { + return nameAndUrls; + } + + public void setNameAndUrls(Map nameAndUrls) { + this.nameAndUrls = nameAndUrls; + } } diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/OnestopMenuVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/OnestopMenuVo.java similarity index 88% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/OnestopMenuVo.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/OnestopMenuVo.java index 214b5d853..e9dadd0cb 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/OnestopMenuVo.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/OnestopMenuVo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,13 +14,11 @@ * */ -package com.webank.wedatasphere.dss.server.dto.response; +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; import java.util.List; -/** - * Created by schumiyi on 2020/6/24 - */ + public class OnestopMenuVo { private Long id; private String title; diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/WorkspaceDepartmentVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/WorkspaceDepartmentVo.java similarity index 83% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/WorkspaceDepartmentVo.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/WorkspaceDepartmentVo.java index 4fc436a33..fb96389d7 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/WorkspaceDepartmentVo.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/WorkspaceDepartmentVo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,11 +14,9 @@ * */ -package com.webank.wedatasphere.dss.server.dto.response; +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; + -/** - * Created by schumiyi on 2020/6/23 - */ public class WorkspaceDepartmentVo { private Long id; diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/WorkspaceFavoriteVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/WorkspaceFavoriteVo.java new file mode 100644 index 000000000..af7d354bc --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/dto/response/WorkspaceFavoriteVo.java @@ -0,0 +1,98 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.dto.response; + +import com.google.common.base.Objects; + + +public class WorkspaceFavoriteVo { + private Long id; + + private Long menuApplicationId; + + private String name; + + private String url; + + private String icon; + + private String title; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Long getMenuApplicationId() { + return menuApplicationId; + } + + public void setMenuApplicationId(Long menuApplicationId) { + this.menuApplicationId = menuApplicationId; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + WorkspaceFavoriteVo that = (WorkspaceFavoriteVo) o; + return Objects.equal(name, that.name); + } + + @Override + public int hashCode() { + return Objects.hashCode(name); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/AbstractDSSVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/AbstractDSSVO.java new file mode 100644 index 000000000..a18bdfd4c --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/AbstractDSSVO.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public abstract class AbstractDSSVO { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSDatasourceVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSDatasourceVO.java new file mode 100644 index 000000000..7121142f4 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSDatasourceVO.java @@ -0,0 +1,123 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.Arrays; + + +public class DSSDatasourceVO { + private int id; + private String name; + private String type; + private String createTime; + private String env; + private String creator; + private String responser; + private String lastUpdater; + private String lastUpdateTime; + + public DSSDatasourceVO() { + } + + public DSSDatasourceVO(int id, String name, String type, String createTime, String env, String creator, String responser, String lastUpdater, String lastUpdateTime) { + this.id = id; + this.name = name; + this.type = type; + this.createTime = createTime; + this.env = env; + this.creator = creator; + this.responser = responser; + this.lastUpdater = lastUpdater; + this.lastUpdateTime = lastUpdateTime; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public String getCreateTime() { + return createTime; + } + + public void setCreateTime(String createTime) { + this.createTime = createTime; + } + + public String getEnv() { + return env; + } + + public void setEnv(String env) { + this.env = env; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public String getResponser() { + return responser; + } + + public void setResponser(String responser) { + this.responser = responser; + } + + public String getLastUpdater() { + return lastUpdater; + } + + public void setLastUpdater(String lastUpdater) { + this.lastUpdater = lastUpdater; + } + + public String getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(String lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } + + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSDictionaryRequestVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSDictionaryRequestVO.java new file mode 100644 index 000000000..2d15d34cb --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSDictionaryRequestVO.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.io.Serializable; +import java.util.List; + +public class DSSDictionaryRequestVO { + + private Integer workspaceId; + + private String dicKey; + + private String parentKey; + + public Integer getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Integer workspaceId) { + this.workspaceId = workspaceId; + } + + public String getDicKey() { + return dicKey; + } + + public void setDicKey(String dicKey) { + this.dicKey = dicKey; + } + + public String getParentKey() { + return parentKey; + } + + public void setParentKey(String parentKey) { + this.parentKey = parentKey; + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSPersonalWorkflowVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSPersonalWorkflowVO.java new file mode 100644 index 000000000..2909ce460 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSPersonalWorkflowVO.java @@ -0,0 +1,83 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.Arrays; +import java.util.List; + + +public class DSSPersonalWorkflowVO extends AbstractDSSVO{ + private int workflowId; + private String name; + private String json; + private String description; + private List tags; + + public DSSPersonalWorkflowVO() { + } + + public DSSPersonalWorkflowVO(int workflowId, String name, String json, String description, List tags) { + this.workflowId = workflowId; + this.name = name; + this.json = json; + this.description = description; + this.tags = tags; + } + + public int getWorkflowId() { + return workflowId; + } + + public void setWorkflowId(int workflowId) { + this.workflowId = workflowId; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getJson() { + return json; + } + + public void setJson(String json) { + this.json = json; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public List getTags() { + return tags; + } + + public void setTags(List tags) { + this.tags = tags; + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSProjectUserVo.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSProjectUserVo.java new file mode 100644 index 000000000..9318f5261 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSProjectUserVo.java @@ -0,0 +1,82 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.Date; + + +public class DSSProjectUserVo { + private int id; + private int projectId; + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public int getProjectId() { + return projectId; + } + + public void setProjectId(int projectId) { + this.projectId = projectId; + } + + public int getUserId() { + return userId; + } + + public void setUserId(int userId) { + this.userId = userId; + } + + public String getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(String workspaceId) { + this.workspaceId = workspaceId; + } + + + + public Date getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(Date lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } + + private int userId; + private String workspaceId; + + public int getPriv_param() { + return priv_param; + } + + public void setPriv_param(int priv_param) { + this.priv_param = priv_param; + } + + private int priv_param; + private Date lastUpdateTime; + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSPublicTableVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSPublicTableVO.java new file mode 100644 index 000000000..563972575 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSPublicTableVO.java @@ -0,0 +1,123 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.Arrays; + + +public class DSSPublicTableVO extends AbstractDSSVO{ + private int id; + private String name; + private String status; + private String type; + private Boolean isPartitionTable; + private String creator; + private String authority; + private String lastOperator; + private String lastOperateTime; + + public DSSPublicTableVO() { + } + + public DSSPublicTableVO(int id, String name, String status, String type, Boolean isPartitionTable, String creator, String authority, String lastOperator, String lastOperateTime) { + this.id = id; + this.name = name; + this.status = status; + this.type = type; + this.isPartitionTable = isPartitionTable; + this.creator = creator; + this.authority = authority; + this.lastOperator = lastOperator; + this.lastOperateTime = lastOperateTime; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public Boolean getPartitionTable() { + return isPartitionTable; + } + + public void setPartitionTable(Boolean partitionTable) { + isPartitionTable = partitionTable; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public String getAuthority() { + return authority; + } + + public void setAuthority(String authority) { + this.authority = authority; + } + + public String getLastOperator() { + return lastOperator; + } + + public void setLastOperator(String lastOperator) { + this.lastOperator = lastOperator; + } + + public String getLastOperateTime() { + return lastOperateTime; + } + + public void setLastOperateTime(String lastOperateTime) { + this.lastOperateTime = lastOperateTime; + } + + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSScheduledWorkFlow.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSScheduledWorkFlow.java new file mode 100644 index 000000000..6fb7e9b7d --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSScheduledWorkFlow.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class DSSScheduledWorkFlow { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSScheduledWorkFlowVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSScheduledWorkFlowVO.java new file mode 100644 index 000000000..0afa18233 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSScheduledWorkFlowVO.java @@ -0,0 +1,131 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.Date; + + +public class DSSScheduledWorkFlowVO extends AbstractDSSVO{ + private String workflowName; + private String project; + private Date scheduledTime; + private String newestVersion; + private String lastScheduledStatus; + private String committer; + private String lastOperator; + private Date lastOperateTime; + private String workflowJson; + private int workflowId; + public DSSScheduledWorkFlowVO() { + } + + public DSSScheduledWorkFlowVO(String workflowName, String project, Date scheduledTime, String newestVersion, + String lastScheduledStatus, String committer, String lastOperator, Date lastOperateTime) { + this.workflowName = workflowName; + this.project = project; + this.scheduledTime = scheduledTime; + this.newestVersion = newestVersion; + this.lastScheduledStatus = lastScheduledStatus; + this.committer = committer; + this.lastOperator = lastOperator; + this.lastOperateTime = lastOperateTime; + } + + public String getWorkflowName() { + return workflowName; + } + + public void setWorkflowName(String workflowName) { + this.workflowName = workflowName; + } + + public String getProject() { + return project; + } + + public void setProject(String project) { + this.project = project; + } + + public Date getScheduledTime() { + return scheduledTime; + } + + public void setScheduledTime(Date scheduledTime) { + this.scheduledTime = scheduledTime; + } + + public String getNewestVersion() { + return newestVersion; + } + + public void setNewestVersion(String newestVersion) { + this.newestVersion = newestVersion; + } + + public String getLastScheduledStatus() { + return lastScheduledStatus; + } + + public void setLastScheduledStatus(String lastScheduledStatus) { + this.lastScheduledStatus = lastScheduledStatus; + } + + public String getCommitter() { + return committer; + } + + public void setCommitter(String committer) { + this.committer = committer; + } + + public String getLastOperator() { + return lastOperator; + } + + public void setLastOperator(String lastOperator) { + this.lastOperator = lastOperator; + } + + public Date getLastOperateTime() { + return lastOperateTime; + } + + public void setLastOperateTime(Date lastOperateTime) { + this.lastOperateTime = lastOperateTime; + } + + public String getWorkflowJson() { + return workflowJson; + } + + public void setWorkflowJson(String workflowJson) { + this.workflowJson = workflowJson; + } + + public int getWorkflowId() { + return workflowId; + } + + public void setWorkflowId(int workflowId) { + this.workflowId = workflowId; + } + + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkflowVersionVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkflowVersionVO.java new file mode 100644 index 000000000..5d13aa5a0 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkflowVersionVO.java @@ -0,0 +1,93 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.Arrays; +import java.util.List; + + +public class DSSWorkflowVersionVO { + private int workflowId; + private String version; + private String creator; + private String comment; + private String createTime; + private String workflowJson; + + public DSSWorkflowVersionVO() { + } + + public DSSWorkflowVersionVO(int workflowId, String version, String creator, String comment, String createTime, String workflowJson) { + this.workflowId = workflowId; + this.version = version; + this.creator = creator; + this.comment = comment; + this.createTime = createTime; + this.workflowJson = workflowJson; + } + + public int getWorkflowId() { + return workflowId; + } + + public void setWorkflowId(int workflowId) { + this.workflowId = workflowId; + } + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + public String getCreateTime() { + return createTime; + } + + public void setCreateTime(String createTime) { + this.createTime = createTime; + } + + public String getWorkflowJson() { + return workflowJson; + } + + public void setWorkflowJson(String workflowJson) { + this.workflowJson = workflowJson; + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceComponentPrivVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceComponentPrivVO.java new file mode 100644 index 000000000..7e217f7fe --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceComponentPrivVO.java @@ -0,0 +1,72 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.io.Serializable; +import java.util.Map; + + +public class DSSWorkspaceComponentPrivVO extends AbstractDSSVO implements Serializable { + private static final long serialVersionUID=1L; + + private int id; + private String name; + private Map componentPrivs; + + + public DSSWorkspaceComponentPrivVO() { + } + + public DSSWorkspaceComponentPrivVO(int id, String name, Map componentPrivs) { + this.id = id; + this.name = name; + this.componentPrivs = componentPrivs; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Map getComponentPrivs() { + return componentPrivs; + } + + public void setComponentPrivs(Map componentPrivs) { + this.componentPrivs = componentPrivs; + } + + @Override + public String toString() { + return "DSSWorkspaceComponentPrivVO{" + + "id=" + id + + ", name='" + name + '\'' + + ", componentPrivs=" + componentPrivs + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceHomePageVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceHomePageVO.java new file mode 100644 index 000000000..35591554e --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceHomePageVO.java @@ -0,0 +1,63 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class DSSWorkspaceHomePageVO { + private int workspaceId; + private String roleName; + private String homePageUrl; + + public DSSWorkspaceHomePageVO() { + } + + public DSSWorkspaceHomePageVO(int workspaceId, String roleName, String username, String homePageUrl) { + this.workspaceId = workspaceId; + this.roleName = roleName; + this.homePageUrl = homePageUrl; + } + + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public String getRoleName() { + return roleName; + } + + public void setRoleName(String roleName) { + this.roleName = roleName; + } + + + + public String getHomePageUrl() { + return homePageUrl; + } + + public void setHomePageUrl(String homePageUrl) { + this.homePageUrl = homePageUrl; + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceHomepageSettingVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceHomepageSettingVO.java new file mode 100644 index 000000000..180e0da6d --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceHomepageSettingVO.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.List; +import java.util.Map; + + +public class DSSWorkspaceHomepageSettingVO { + + public static class RoleHomepage{ + private String roleName; + private String homepageName; + private String homepageUrl; + private int roleId; + private String roleFrontName; + + + public String getRoleName() { + return roleName; + } + + public void setRoleName(String roleName) { + this.roleName = roleName; + } + + public String getHomepageName() { + return homepageName; + } + + public void setHomepageName(String homepageName) { + this.homepageName = homepageName; + } + + public String getHomepageUrl() { + return homepageUrl; + } + + public void setHomepageUrl(String homepageUrl) { + this.homepageUrl = homepageUrl; + } + + + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public String getRoleFrontName() { + return roleFrontName; + } + + public void setRoleFrontName(String roleFrontName) { + this.roleFrontName = roleFrontName; + } + } + + private List roleHomepages; + + public List getRoleHomepages() { + return roleHomepages; + } + + public void setRoleHomepages(List roleHomepages) { + this.roleHomepages = roleHomepages; + } + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuPrivVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuPrivVO.java new file mode 100644 index 000000000..ed4c86620 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuPrivVO.java @@ -0,0 +1,70 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.io.Serializable; +import java.util.Map; + + +public class DSSWorkspaceMenuPrivVO extends AbstractDSSVO implements Serializable { + private static final long serialVersionUID=1L; + private int id; + private String name; + private Map menuPrivs; + + public DSSWorkspaceMenuPrivVO() { + } + + public DSSWorkspaceMenuPrivVO(int id, String name, Map menuPrivs) { + this.id = id; + this.name = name; + this.menuPrivs = menuPrivs; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Map getMenuPrivs() { + return menuPrivs; + } + + public void setMenuPrivs(Map menuPrivs) { + this.menuPrivs = menuPrivs; + } + + @Override + public String toString() { + return "DSSWorkspaceMenuPrivVO{" + + "id=" + id + + ", name='" + name + '\'' + + ", menuPrivs=" + menuPrivs + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuPrivsVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuPrivsVO.java new file mode 100644 index 000000000..85e513b74 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuPrivsVO.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class DSSWorkspaceMenuPrivsVO { + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public String getRoleName() { + return roleName; + } + + public void setRoleName(String roleName) { + this.roleName = roleName; + } + + public Boolean getVisable() { + return isVisable; + } + + public void setVisable(Boolean visable) { + isVisable = visable; + } + + private int roleId; + private String roleName; + private Boolean isVisable; +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuVO.java new file mode 100644 index 000000000..22055243b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceMenuVO.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.ArrayList; +import java.util.List; + + +public class DSSWorkspaceMenuVO { + private int id; + private String name; + private String icon; + private List subMenus; + + public DSSWorkspaceMenuVO(){ + + } + + public DSSWorkspaceMenuVO(int id, String name, List subMenus) { + this.id = id; + this.name = name; + this.subMenus = subMenus; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public List getSubMenus() { + return subMenus; + } + + public void setSubMenus(List subMenus) { + this.subMenus = subMenus; + } + + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceOverviewVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceOverviewVO.java new file mode 100644 index 000000000..885b20493 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceOverviewVO.java @@ -0,0 +1,119 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.List; +import java.util.Map; + + +public class DSSWorkspaceOverviewVO { + private String title; + + + private String description; + + private String dssDescription; + + + + + private List videos; + + + Map> demos; + + public static class OverviewInfo{ + private int id; + private String title; + private String url; + + public OverviewInfo(){ + + } + + public OverviewInfo(String title, String url){ + this.title = title; + this.url = url; + } + + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getDssDescription() { + return dssDescription; + } + + public void setDssDescription(String dssDescription) { + this.dssDescription = dssDescription; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + + public Map> getDemos() { + return demos; + } + + public void setDemos(Map> demos) { + this.demos = demos; + } + + public List getVideos() { + return videos; + } + + public void setVideos(List videos) { + this.videos = videos; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspacePrivVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspacePrivVO.java new file mode 100644 index 000000000..fe5e3771d --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspacePrivVO.java @@ -0,0 +1,88 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.io.Serializable; +import java.util.List; + + +public class DSSWorkspacePrivVO implements Serializable { + private static final long serialVersionUID=1L; + private int workspaceId; + private List menuPrivVOS; + private List componentPrivVOS; + private List roleVOS; + + public List getRoleVOS() { + return roleVOS; + } + + public void setRoleVOS(List roleVOS) { + this.roleVOS = roleVOS; + } + + + + + public DSSWorkspacePrivVO() { + } + + public DSSWorkspacePrivVO(int workspaceId, List menuPrivVOS, List componentPrivVOS,List roleVOS) { + this.workspaceId = workspaceId; + this.menuPrivVOS = menuPrivVOS; + this.componentPrivVOS = componentPrivVOS; + this.roleVOS=roleVOS; + } + + public int getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(int workspaceId) { + this.workspaceId = workspaceId; + } + + public List getMenuPrivVOS() { + return menuPrivVOS; + } + + public void setMenuPrivVOS(List menuPrivVOS) { + this.menuPrivVOS = menuPrivVOS; + } + + public List getComponentPrivVOS() { + return componentPrivVOS; + } + + public void setComponentPrivVOS(List componentPrivVOS) { + this.componentPrivVOS = componentPrivVOS; + } + + @Override + public String toString() { + return "DSSWorkspacePrivVO{" + + "workspaceId=" + workspaceId + + ", menuPrivVOS=" + menuPrivVOS + + ", componentPrivVOS=" + componentPrivVOS + + ", roleVOS=" + roleVOS + + '}'; + } + + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceRoleVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceRoleVO.java new file mode 100644 index 000000000..3e0d681df --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceRoleVO.java @@ -0,0 +1,62 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.io.Serializable; + + +public class DSSWorkspaceRoleVO implements Serializable { + private static final long serialVersionUID=1L; + + private int roleId; + private String roleName; + private String roleFrontName; + + + public int getRoleId() { + return roleId; + } + + public void setRoleId(int roleId) { + this.roleId = roleId; + } + + public String getRoleName() { + return roleName; + } + + public void setRoleName(String roleName) { + this.roleName = roleName; + } + + public String getRoleFrontName() { + return roleFrontName; + } + + public void setRoleFrontName(String roleFrontName) { + this.roleFrontName = roleFrontName; + } + + @Override + public String toString() { + return "DSSWorkspaceRoleVO{" + + "roleId=" + roleId + + ", roleName='" + roleName + '\'' + + ", roleFrontName='" + roleFrontName + '\'' + + '}'; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceScriptisStatiticVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceScriptisStatiticVO.java new file mode 100644 index 000000000..8331e46b0 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceScriptisStatiticVO.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.HashMap; +import java.util.Map; + + +public class DSSWorkspaceScriptisStatiticVO extends DSSWorkspaceStatisticVO{ + private String name; + + private String status; + + private Map statistics; + + public DSSWorkspaceScriptisStatiticVO() { + } + + public DSSWorkspaceScriptisStatiticVO(String name, String status, Map statistics) { + this.name = name; + this.status = status; + this.statistics = statistics; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public Map getStatistics() { + return statistics; + } + + public void setStatistics(Map statistics) { + this.statistics = statistics; + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceStatisticVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceStatisticVO.java new file mode 100644 index 000000000..259a97740 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceStatisticVO.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class DSSWorkspaceStatisticVO extends AbstractDSSVO{ + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceUserVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceUserVO.java new file mode 100644 index 000000000..a3aa5775f --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceUserVO.java @@ -0,0 +1,97 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + + +public class DSSWorkspaceUserVO extends AbstractDSSVO{ + private int id; + private String name; + private List roles; + private String department; + private String office; + private String creator; + private Date joinTime; + + public DSSWorkspaceUserVO() { + } + + + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public List getRoles() { + return roles; + } + + public void setRoles(List roles) { + this.roles = roles; + } + + public String getDepartment() { + return department; + } + + public void setDepartment(String department) { + this.department = department; + } + + public String getOffice() { + return office; + } + + public void setOffice(String office) { + this.office = office; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Date getJoinTime() { + return joinTime; + } + + public void setJoinTime(Date joinTime) { + this.joinTime = joinTime; + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceVO.java new file mode 100644 index 000000000..c5f90b30a --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DSSWorkspaceVO.java @@ -0,0 +1,89 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class DSSWorkspaceVO extends AbstractDSSVO{ + + + private int id; + private String name; + private String tags; + private String department; + private String description; + private String product; + + + public DSSWorkspaceVO() { + } + + public DSSWorkspaceVO(int id, String name){ + this.id = id; + this.name = name; + } + + + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getTags() { + return tags; + } + + public void setTags(String tags) { + this.tags = tags; + } + + + public String getDepartment() { + return department; + } + + public void setDepartment(String department) { + this.department = department; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getProduct() { + return product; + } + + public void setProduct(String product) { + this.product = product; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DepartmentVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DepartmentVO.java new file mode 100644 index 000000000..3e5a70582 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/DepartmentVO.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class DepartmentVO { + private int id; + private String name; + private String frontName; + + public DepartmentVO(){ + + } + + public DepartmentVO(int id, String name, String frontName) { + this.id = id; + this.name = name; + this.frontName = frontName; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getFrontName() { + return frontName; + } + + public void setFrontName(String frontName) { + this.frontName = frontName; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SecondaryWorkspaceMenuVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SecondaryWorkspaceMenuVO.java new file mode 100644 index 000000000..e8cd562c4 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SecondaryWorkspaceMenuVO.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class SecondaryWorkspaceMenuVO { + private int id; + private String name; + private String url; + + public SecondaryWorkspaceMenuVO() { + } + + public SecondaryWorkspaceMenuVO(int id, String name, String url) { + this.id = id; + this.name = name; + this.url = url; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SidebarContentVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SidebarContentVO.java new file mode 100644 index 000000000..694b93a42 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SidebarContentVO.java @@ -0,0 +1,74 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +public class SidebarContentVO { + + //名称 + private String name; + //标题 + private String title; + + private String url; + /** + * url类型: 0-内部系统，1-外部系统；默认是内部 + */ + private Integer urlType; + /** + * icon是表示Content的图标，如果为空就是没有 + */ + private String icon; + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getUrl() { + return url; + } + + public void setUrl(String url) { + this.url = url; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public Integer getUrlType() { + return urlType; + } + + public void setUrlType(Integer urlType) { + this.urlType = urlType; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SidebarVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SidebarVO.java new file mode 100644 index 000000000..a9dfb8f35 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/SidebarVO.java @@ -0,0 +1,64 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import java.util.List; + +public class SidebarVO { + + + private String name; + + + private String title; + + private Integer type; + + private List contents; + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public Integer getType() { + return type; + } + + public void setType(Integer type) { + this.type = type; + } + + public List getContents() { + return contents; + } + + public void setContents(List contents) { + this.contents = contents; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/StaffInfoVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/StaffInfoVO.java new file mode 100644 index 000000000..caeb7a8e6 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/StaffInfoVO.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + + +public class StaffInfoVO { + private String id; + private String username; + private String department; + private String office; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public String getDepartment() { + return department; + } + + public void setDepartment(String department) { + this.department = department; + } + + public String getOffice() { + return office; + } + + public void setOffice(String office) { + this.office = office; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/VOUtils.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/VOUtils.java new file mode 100644 index 000000000..bc70f999a --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/VOUtils.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo; + +import com.google.gson.Gson; +import com.google.gson.GsonBuilder; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; + + +public class VOUtils { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/AbstractWorkspaceComponentInfoVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/AbstractWorkspaceComponentInfoVO.java new file mode 100644 index 000000000..cd67ffe35 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/AbstractWorkspaceComponentInfoVO.java @@ -0,0 +1,131 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo.info; + +import java.util.List; + + +public abstract class AbstractWorkspaceComponentInfoVO { + + private String title; + + private String icon; + + private String desc; + + private String buttonText; + + private List statistics; + + private String componentUrl; + + private String userManualUrl; + + private String indicatorUrl; + + public String getTitle() { + return title; + } + + public static class ComponentStatistic{ + private String name; + private String value; + + public ComponentStatistic(String name, String value) { + this.name = name; + this.value = value; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getValue() { + return value; + } + + public void setValue(String value) { + this.value = value; + } + } + + + public void setTitle(String title) { + this.title = title; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public String getDesc() { + return desc; + } + + public void setDesc(String desc) { + this.desc = desc; + } + + public String getButtonText() { + return buttonText; + } + + public void setButtonText(String buttonText) { + this.buttonText = buttonText; + } + + + public List getStatistics() { + return statistics; + } + + public void setStatistics(List statistics) { + this.statistics = statistics; + } + + public String getComponentUrl() { + return componentUrl; + } + + public void setComponentUrl(String componentUrl) { + this.componentUrl = componentUrl; + } + + public String getUserManualUrl() { + return userManualUrl; + } + + public void setUserManualUrl(String userManualUrl) { + this.userManualUrl = userManualUrl; + } + + public String getIndicatorUrl() { + return indicatorUrl; + } + + public void setIndicatorUrl(String indicatorUrl) { + this.indicatorUrl = indicatorUrl; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceScriptisInfoVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceScriptisInfoVO.java new file mode 100644 index 000000000..b161dbdc1 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceScriptisInfoVO.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo.info; + + +public class DSSWorkspaceScriptisInfoVO extends AbstractWorkspaceComponentInfoVO { + + private static final String SCRIPTIS_DESC = "Scriptis是微众银行微数域(WeDataSphere)打造的一站式交互式数据探索分析工具，以任意桥(Linkis)做为内核，提供多种计算存储引擎(如Spark、Hive、TiSpark等)、Hive数据库管理功能、资源(如Yarn资源、服务器资源)管理、应用管理和各种用户资源(如UDF、变量等)管理的能力。"; + + private static final String SCRIPTIS_BUTTON_TEXT = "进入Scriptis"; + + private static final String SCRIPTIS_ICON = "fi-scriptis"; + + private static final String SCRIPTIS_MANUAL_URL = "http://127.0.0.1:8088/wiki/scriptis/manual/feature_overview_cn.html"; + + private static final String SCRIPTIS_TITLE = "Scriptis"; + + public DSSWorkspaceScriptisInfoVO() { + this.setTitle(SCRIPTIS_TITLE); + this.setButtonText(SCRIPTIS_BUTTON_TEXT); + this.setDesc(SCRIPTIS_DESC); + this.setIcon(SCRIPTIS_ICON); + this.setUserManualUrl(SCRIPTIS_MANUAL_URL); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceVisualisInfoVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceVisualisInfoVO.java new file mode 100644 index 000000000..d11c1c7a3 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceVisualisInfoVO.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo.info; + + +public class DSSWorkspaceVisualisInfoVO extends AbstractWorkspaceComponentInfoVO{ + + private static final String VISUALIS_DESC = "Visualis是一个可视化BI工具，支持拖拽式报表定义、图表联动、钻取、全局筛选、多维分析、实时查询等数据开发探索的可视化分析模式。"; + + private static final String VISUALIS_BUTTON_TEXT = "进入Visualis"; + + private static final String VISUALIS_ICON = "fi-visualis"; + + private static final String VISUALIS_MANUAL_URL = "http://127.0.0.1:8088/wiki/scriptis/manual/feature_overview_cn.html"; + + private static final String VISUALIS_TITLE = "Visualis"; + + public DSSWorkspaceVisualisInfoVO() { + this.setTitle(VISUALIS_TITLE); + this.setButtonText(VISUALIS_BUTTON_TEXT); + this.setDesc(VISUALIS_DESC); + this.setIcon(VISUALIS_ICON); + this.setUserManualUrl(VISUALIS_MANUAL_URL); + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceWorkflowInfoVO.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceWorkflowInfoVO.java new file mode 100644 index 000000000..04dedf499 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/bean/vo/info/DSSWorkspaceWorkflowInfoVO.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.bean.vo.info; + + +public class DSSWorkspaceWorkflowInfoVO extends AbstractWorkspaceComponentInfoVO{ + + //private static final String WORKFLOW_DESC = "DSS系统工作流开发是支持以可视化编辑方式进行工作流编排，" + + //"同时支持工作流实时执行并能够一键发布到WTSS进行定时调度的组件，工作流开发已经支持Scriptis、Visualis以及Qualitis等子系统的接入。"; + + private static final String WORKFLOW_DESC = "DataSphere Studio workflow development is a component which supports workflow editing " + + "through visual editing, and supports real-time execution of workflows and releasing workflows to WTSS by one-click scheduling as well. " + + "Now it has supported the connection of subsystems such as Scriptis, Visualis, and Qualitis etc. "; + + private static final String WORKFLOW_BUTTON_TEXT = "Enter Workflow Development"; + + private static final String WORKFLOW_ICON = "fi-workflow"; + + private static final String WORKFLOW_MANUAL_URL = "http://127.0.0.1:8088/wiki/scriptis/manual/feature_overview_cn.html"; + + private static final String WORKFLOW_TITLE = "Workflow Development"; + + public DSSWorkspaceWorkflowInfoVO() { + this.setTitle(WORKFLOW_TITLE); + this.setButtonText(WORKFLOW_BUTTON_TEXT); + this.setDesc(WORKFLOW_DESC); + this.setIcon(WORKFLOW_ICON); + this.setUserManualUrl(WORKFLOW_MANUAL_URL); + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/constant/ApplicationConf.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/constant/ApplicationConf.java new file mode 100644 index 000000000..196a59d9e --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/constant/ApplicationConf.java @@ -0,0 +1,42 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.constant; + + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public class ApplicationConf { + + public static final CommonVars FAQ = CommonVars.apply("wds.linkis.application.dws.params","http://127.0.0.1:8088/wiki/scriptis/manual/feature_overview_cn.html"); + public static final CommonVars SCHEDULIS_URL = + CommonVars.apply("wds.linkis.schedulis.url", "http://127.0.0.1:8088"); + + public static final CommonVars HOMEPAGE_MODULE_NAME = + CommonVars.apply("wds.linkis.special.homepage.module.name", "apiServices"); + + public static final CommonVars HOMEPAGE_URL = + CommonVars.apply("wds.linkis.special.homepage.module.url", "/newHome?workspaceId="); + + public static final CommonVars DSS_ENV_PROD_LABEL = + CommonVars.apply("wds.dss.env.prod.label", "PROD"); + + public static final String SCHEDULER_APP_CONN_NAME = CommonVars.apply("wds.dss.appconn.scheduler.name", "schedulis").getValue(); + + public static final String ESB_APPID = CommonVars.apply("wds.dss.esb.appid", "").getValue(); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSApplicationMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSApplicationMapper.java new file mode 100644 index 000000000..031ff8714 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSApplicationMapper.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSApplication; +import com.webank.wedatasphere.dss.framework.workspace.bean.Sidebar; +import org.apache.ibatis.annotations.Mapper; + + +@Mapper +public interface DSSApplicationMapper extends BaseMapper { + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSComponentRoleMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSComponentRoleMapper.java new file mode 100644 index 000000000..a52fb355b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSComponentRoleMapper.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSComponentRole; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceComponentPriv; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceHomepage; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface DSSComponentRoleMapper extends BaseMapper { + + + public void insertBatch(@Param("list") List list); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSDictionaryMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSDictionaryMapper.java new file mode 100644 index 000000000..e120caec7 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSDictionaryMapper.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSDictionary; +import org.apache.ibatis.annotations.Mapper; + + +@Mapper +public interface DSSDictionaryMapper extends BaseMapper { + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSMenuRoleMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSMenuRoleMapper.java new file mode 100644 index 000000000..d7f215aae --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSMenuRoleMapper.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceMenuRole; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSMenuRole; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface DSSMenuRoleMapper extends BaseMapper { + + public void insertBatch(@Param("list") List menuRoles); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSUserMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSUserMapper.java new file mode 100644 index 000000000..1f8375d47 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSUserMapper.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSUser; +import org.apache.ibatis.annotations.Mapper; + + +@Mapper +public interface DSSUserMapper extends BaseMapper { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceHomepageMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceHomepageMapper.java new file mode 100644 index 000000000..c183ad978 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceHomepageMapper.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceHomepage; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceMenuRole; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface DSSWorkspaceHomepageMapper extends BaseMapper { + + public void insertBatch(@Param("list") List homepageList); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceInfoMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceInfoMapper.java new file mode 100644 index 000000000..a1f6807e8 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceInfoMapper.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspace; +import org.apache.ibatis.annotations.*; + + +@Mapper +public interface DSSWorkspaceInfoMapper { + + @Select("select name from dss_workspace where id = #{workspaceId} ") + String getWorkspaceNameById(@Param("workspaceId") int workspaceId); + + @Select("select id from dss_workspace where name = #{workspaceName}") + int getWorkspaceIdByName(@Param("workspaceName") String workspaceName); + + @Select("select * from dss_workspace where id = #{workspaceId}") + @Results({ + @Result(property = "createBy", column = "create_by"), + @Result(property = "createTime", column = "create_time"), + @Result(property = "lastUpdateTime", column = "last_update_time"), + @Result(property = "lastUpdateUser", column = "last_update_user"), + }) + DSSWorkspace getWorkspace(@Param("workspaceId") int workspaceId); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceMapper.java new file mode 100644 index 000000000..a4e0adc71 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceMapper.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.webank.wedatasphere.dss.framework.workspace.bean.*; +import org.apache.ibatis.annotations.Insert; +import org.apache.ibatis.annotations.Mapper; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +@Mapper +public interface DSSWorkspaceMapper { + + void createWorkSpace(DSSWorkspace dssWorkspace); + + List getWorkspaces(String username); + + List getMenuId(int roleId, String workspaceId); + + DSSMenu getSpaceMenu(int menuId); + + List getDSSWorkspaceMenuPriv(String workspaceId); + + @Insert({ + "" + }) + void setDefaultComponentRoles(@Param("privs") List dssWorkspaceComponentPrivs); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceMenuMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceMenuMapper.java new file mode 100644 index 000000000..10f2e07f6 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceMenuMapper.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceComponentRolePriv; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceHomepageSetting; +import org.apache.ibatis.annotations.*; + +import java.util.List; + + +@Mapper +public interface DSSWorkspaceMenuMapper { + + + + @Select("select url from dss_menu_component_url where menu_id = #{menuId}") + String getSubMenuUrl(@Param("menuId") int menuId); + + + @Select("select * from dss_component_role where workspace_id = #{workspaceId}") + @Results({ + @Result(property = "workspaceId", column = "workspace_id"), + @Result(property = "componentId", column = "component_id"), + @Result(property = "roleId", column = "role_id"), + @Result(property = "lastUpdateTime", column = "last_update_time"), + @Result(property = "updateBy", column = "update_by") + }) + List getComponentRolePriv(@Param("workspaceId") int workspaceId); + + @Select("select * from dss_workspace_homepage where workspace_id = #{workspaceId}") + @Results({ + @Result(property = "workspaceId", column = "workspace_id"), + @Result(property = "homepageUrl", column = "homepage_url"), + @Result(property = "roleId", column = "role_id"), + @Result(property = "updateTime", column = "update_time") + }) + List getWorkspaceHompageSettings(@Param("workspaceId") int workspaceId); + + @Select("select priv from dss_component_role where workspace_id = #{workspaceId} " + + "and role_id = #{roleId} and " + + "component_id = #{applicationId}") + Integer getOneCompoentRolePriv(@Param("workspaceId") int workspaceId, + @Param("roleId") int roleId, @Param("applicationId") int applicationId); + + /** + * 如果在dss_component_role中是有workspace_id=-1的情况, + * 默认是-1的时候，就要全部拿出来 + * @return + */ + @Select("select * from dss_component_role where workspace_id = -1") + @Results({ + @Result(property = "workspaceId", column = "workspace_id"), + @Result(property = "componentId", column = "component_id"), + @Result(property = "roleId", column = "role_id"), + @Result(property = "lastUpdateTime", column = "last_update_time"), + @Result(property = "updateBy", column = "update_by") + }) + List getDefaultComponentRolePriv(); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspacePrivMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspacePrivMapper.java new file mode 100644 index 000000000..5938af568 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspacePrivMapper.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import org.apache.ibatis.annotations.Mapper; +import org.apache.ibatis.annotations.Param; +import org.apache.ibatis.annotations.Select; +import org.apache.ibatis.annotations.Update; + + +@Mapper +public interface DSSWorkspacePrivMapper { + + @Update("update dss_menu_role set priv = #{priv}, update_time = now() " + + "where workspace_id = #{workspaceId} and menu_id = #{menuId} and role_id = #{roleId}") + void updateRoleMenuPriv(@Param("workspaceId") int workspaceId, @Param("menuId") int menuId, + @Param("roleId") int roleId, @Param("priv") int priv); + + @Update("update dss_component_role set priv = #{priv} , update_time = now()" + + "where workspace_id = #{workspaceId} and component_id = #{componentId} and role_id = #{roleId}") + void updateRoleComponentPriv(@Param("workspaceId") int workspaceId, @Param("componentId") int componentId, + @Param("roleId") int roleId, @Param("priv") int priv); + + @Select("select id from dss_role where workspace_id = #{workspaceId} and name = #{key}") + Integer getRoleId(@Param("workspaceId") int workspaceId, @Param("key") String key); + + @Select("select count(*) from dss_component_role where workspace_id = #{workspaceId} and component_id = #{componentId} and role_id = #{roleId}") + int queryCntOfRCP(@Param("workspaceId") int workspaceId, @Param("componentId") int componentId, @Param("roleId") int roleId); + + @Select("insert into dss_component_role (`workspace_id`, `component_id`, `role_id`, `priv`, `update_time`, `updateby`) " + + "values(#{workspaceId}, #{componentId}, #{roleId}, #{priv}, now(), 'cooperyang')") + void insertRolComponentPriv(@Param("workspaceId") int workspaceId, @Param("componentId") int componentId, @Param("roleId") int roleId, + @Param("priv") int priv); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceProjectMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceProjectMapper.java new file mode 100644 index 000000000..03073c9fa --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceProjectMapper.java @@ -0,0 +1,157 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import org.apache.ibatis.annotations.*; + +import java.util.Date; +import java.util.List; + + +@Mapper +public interface DSSWorkspaceProjectMapper { + + + + + + /*@Select("select * from dss_project where workspace_id = #{workspaceId} and name = #{projectName}") + @Results(id ="dss_project_map", value = { + @Result(property = "workspaceId", column = "workspace_id"), + @Result(property = "isPersonal", column = "is_personal"), + @Result(property = "id", column = "id"), + @Result(property = "updateTime",column = "update_time") + }) + DSSPersonalProject searchPersonalProjectInWorkspace(@Param("workspaceId") int workspaceId, + @Param("username") String username, @Param("projectName") String projectName);*/ + + + @Insert({ + "" + }) + void setProjectPriv(@Param("workspaceId") int workspaceId, @Param("projectId") Long projectId, + @Param("usernames") List usernames, @Param("priv") int priv, @Param("lastUpdateTime") Date date); + + + + /*@Select("select * from dss_project where id in (" + + "select project_id from dss_project_user where " + + "workspace_id = #{workspaceId} and username = #{username} and priv >= 1" + + ") and is_personal = 0 or username = #{username} and workspace_id = #{workspaceId} and is_personal = 0") + @ResultMap(value = "dss_project_map") + List listCooperateProjects(@Param("workspaceId") int workspaceId, @Param("username") String username);*/ + + @Select("select priv from dss_project_user where workspace_id = #{workspaceId} " + + "and username = #{username} " + + "and project_id = #{projectId}") + int selectPriv(@Param("projectId") Long projectId, @Param("workspaceId") int workspaceId, @Param("username") String username); + + + /*@Select("select * from dss_flow where project_id in " + + "(select distinct project_id from dss_project_user " + + "where workspace_id = #{workspaceId} and priv >=1 and username = #{username}) " + + "or id in " + + "(select flow_id from dss_flow_user where workspace_id = #{workspaceId} and priv >= 1 " + + "and username = #{username})") + @Results({ + @Result(property = "creatorID", column = "creator_id"), + @Result(property = "createTime", column = "create_time"), + @Result(property = "projectID", column = "project_id"), + @Result(property = "isRootFlow", column = "is_root_flow") + }) + List getReleasedWorkflows(@Param("workspaceId") int workspaceId, + @Param("username") String username); +*/ + @Select("select name from dss_project where id = #{projectId}") + String getProjectNameById(@Param("projectId") Long projectId); + + @Select("select max(version) from dss_flow_version where flow_id = #{flowId}") + String getFlowLatestVersion(@Param("flowId") Long flowId); + + @Select("select username from dss_flow_user where flow_id = #{flowId} and project_id = #{projectId} and workspace_id = #{workspaceId}") + List getAccessUsersByFlowId(@Param("flowId") Long flowId, @Param("projectId") Long projectId, + @Param("workspaceId") int workspaceId); + + @Delete("delete from dss_flow_user where workspace_id = #{workspaceId} and project_id = #{projectId} and flow_id = #{flowId}") + void deleteAllWorkflowPriv(@Param("workspaceId") int workspaceId, + @Param("projectId") int projectId, + @Param("flowId") int flowId); + + @Delete("delete from dss_project_user where workspace_id = #{workspaceId} and project_id = #{projectId}") + void deleteAllProjectPriv(@Param("workspaceId") int workspaceId, @Param("projectId") int projectId); + + @Insert({ + "" + }) + void setWorkflowPriv(@Param("workspaceId") int workspaceId, + @Param("projectId") int projectId, + @Param("flowId") int flowId, + @Param("accessUsers") List accessUsers, + @Param("priv") int priv, + @Param("updateTime") Date updateTime); + + @Select("select name from dss_flow where id = #{flowId}") + String getFlowNameById(@Param("flowId") int flowId); + + @Select("select id from dss_project where name = #{projectName}") + int getProjectIdByName(@Param("projectName") String projectName); + + /* @Select("select * from dss_flow_schedule_info where flow_id = #{flowId}") + @Results({ + @Result(property = "scheduleTime", column = "schedule_time"), + @Result(property = "alarmLevel", column = "alarm_level"), + @Result(property = "alarmUserEmails", column = "alarm_user_emails") + }) + DSSReleasedFlowVO.ScheduleInfo getScheduleInfo(@Param("flowId") Long flowId); +*/ + + + @Insert("insert into dss_flow_schedule_info(flow_id, schedule_time, alarm_user_emails, alarm_level) " + + "values(#{flowId},#{scheduleTime}, #{alarmEmails}, #{alarmLevel})") + void setScheduleInfo(@Param("scheduleTime") String scheduleTime, @Param("alarmEmails") String alarmEmails, + @Param("alarmLevel") String alarmLevel, @Param("flowId") int flowId); + + @Delete("delete from dss_flow_schedule_info where flow_id = #{flowId}") + void deleteScheduleInfo(@Param("flowId") int flowId); + + @Select("select username from dss_project_user WHERE project_id = #{projectId} and priv = 1") + List getAccessUsersByProjectId(@Param("projectId") Long projectId); + + + @Select("select username from dss_project_user WHERE project_id = #{projectId} and priv = 2") + List getEditUsersByProjectId(Long id); + + @Select("select project_id from dss_project_version where id = #{projectVersionId}") + Long getProjectIdByVersionId(@Param("projectVersionId") Long projectVersionId); + + @Update("update dss_project set workspace_id = #{workspaceId} where id = #{projectId}") + void setWorkspaceIdForProject(@Param("workspaceId") Integer workspaceId, @Param("projectId") Long projectId); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceRoleMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceRoleMapper.java new file mode 100644 index 000000000..2db75f9f0 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceRoleMapper.java @@ -0,0 +1,138 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.webank.wedatasphere.dss.framework.workspace.bean.*; +import org.apache.ibatis.annotations.*; + +import java.util.List; + + +@Mapper +public interface DSSWorkspaceRoleMapper { + + @Select("select * from dss_role") + @Results({ + @Result(property = "workspaceId", column = "workspace_id"), + @Result(property = "frontName", column = "front_name"), + @Result(property = "createTime", column = "update_time") + }) + List getRoles(); + + + @Select("select a.`name` from dss_role a join dss_workspace_user_role b on a.id = b.role_id" + + " and b.`username` = #{username} and b.workspace_id = #{workspaceId}") + List getAllRoles(@Param("username") String username, @Param("workspaceId") int workspaceId); + + + @Select("select * from dss_menu") + @Results({ + @Result(property = "upperMenuId", column = "upper_menu_id"), + @Result(property = "frontName", column = "front_name"), + @Result(property = "isActive", column = "is_active") + }) + List getMenus(); + + + @Select("select * from dss_menu_component_url") + @Results({ + @Result(property = "menuId", column = "menu_id"), + @Result(property = "dssApplicationId", column = "dss_application_id"), + @Result(property = "manulUrl", column = "manul_url"), + @Result(property = "operationUrl", column = "operation_url"), + @Result(property = "updateTime", column = "update_time"), + }) + List getMenuComponentUrl(); + + + @Select("select homepage_url from dss_application where id = #{applicationId}") + String getEntryUrl(@Param("applicationId") int applicationId); + + + + @Select("select * from dss_application") + @Results(id = "dss_application_map", value = { + @Result(property = "id", column = "id"), + @Result(property = "homepageUrl", column = "hoempage_url"), + @Result(property = "componentName", column = "name") + }) + List getComponents(); + + + @Select("select * from dss_application") + @Results(value = { + @Result(property = "id", column = "id"), + @Result(property = "name", column = "name"), + @Result(property = "homepageUrl", column = "homepage_url"), + @Result(property = "projectUrl", column = "project_url") + }) + List getDSSApplications(); + + @Insert("insert into dss_role(workspace_id, name, front_name, update_time) " + + "values(#{dssRole.workspaceId}, #{dssRole.name}, #{dssRole.frontName}, #{dssRole.createTime})") + @Options(useGeneratedKeys = true, keyProperty = "dssRole.id", keyColumn = "id") + int addNewRole(@Param("dssRole") DSSRole dssRole); + + + @Insert({ + "" + }) + void updateRoleMenu(@Param("roleId") int roleId, @Param("workspaceId") int workspaceId, + @Param("menuIds") List menuIds, @Param("username") String username, + @Param("priv") Integer priv); + + + + @Insert({ + "" + }) + void updateRoleComponent(@Param("roleId") int roleId, @Param("workspaceId") int workspaceId, + @Param("componentIds") List componentIds, @Param("username") String username, + @Param("priv") Integer priv); + + + @Select("select workspace_id from dss_workspace_user where username = #{username}") + List getWorkspaceIds(@Param("username") String username); + + @Select("select workspace_id from dss_workspace where `name` = #{defaultWorkspaceName}") + Integer getDefaultWorkspaceId(@Param("defaultWorkspaceName") String defaultWorkspaceName); + + @Select("select id from dss_role where workspace_id = #{workspaceId} and name = #{apiUser}") + int getRoleId(@Param("apiUser") String apiUser, @Param("workspaceId") int workspaceId); + + @Select("Select count(*) from dss_component_role where workspace_id = #{workspaceId} and role_id = #{roleId} and component_id = #{componentId}") + int getCount(@Param("workspaceId") Integer workspaceId, @Param("componentId") int componentId, @Param("roleId") int roleId); + + @Select("select priv from dss_component_role where workspace_id = #{workspaceId} and role_id = #{roleId} and " + + "component_id = #{componentId}") + Integer getPriv(@Param("workspaceId") Integer workspaceId, @Param("roleId") int roleId, @Param("componentId") int componentId); + + @Select("select id from dss_application where `name` = #{appName}") + int getComponentId(@Param("appName") String appName); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceUserMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceUserMapper.java new file mode 100644 index 000000000..7f574d466 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/DSSWorkspaceUserMapper.java @@ -0,0 +1,99 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSUser; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceUser; +import org.apache.ibatis.annotations.*; + +import java.util.List; + + +@Mapper +public interface DSSWorkspaceUserMapper { + + Long getUserID(String userName); + + String getUserName(Long userID); + + @Insert("insert into dss_workspace_user(workspace_id, username, join_time, created_by)" + + "values(#{workspaceId}, #{username}, now(), #{creator})") + void insertUser(@Param("username") String username, + @Param("workspaceId") int workspaceId, @Param("creator") String creator); + + + @Insert("insert into dss_workspace_user_role(workspace_id, username, role_id, create_time, created_by)" + + "values(#{workspaceId}, #{username}, #{roleId}, now(), #{createdBy})") + void setUserRoleInWorkspace(@Param("workspaceId") int workspaceId, @Param("roleId") int roleId, + @Param("username") String username, @Param("createdBy") String createdBy); + + @Select("select role_id from dss_workspace_user_role where workspace_id = #{workspaceId} and username = #{username}") + List getRoleInWorkspace(@Param("workspaceId") int workspaceId, @Param("username") String username); + + + @Delete("delete from dss_workspace_user_role where username = #{username} and workspace_id = #{workspaceId}") + void removeAllRolesForUser(@Param("username") String username, @Param("workspaceId") int workspaceId); + + @Delete("delete from dss_workspace_user where username = #{username} and workspace_id = #{workspaceId}") + void removeUserInWorkspace(@Param("username") String username, @Param("workspaceId") int workspaceId); + + @Select("select workspace_id from dss_workspace_user where username = #{username}") + List getWorkspaceIds(@Param("username") String username); + + @Select("select homepage_url from dss_workspace_homepage where workspace_id = #{workspaceId} and role_id = #{roleId}") + String getHomepageUrl(@Param("workspaceId") int workspaceId, @Param("roleId") int roleId); + + @Select("select * from dss_user") + List listAllDSSUsers(); + + @Select("select username from dss_workspace_user where workspace_id = #{workspaceId}") + List getAllWorkspaceUsers(@Param("workspaceId") int workspaceId); + + @Select("select username from dss_flow_user where flow_id = #{flowId}") + List getFlowUser(@Param("flowId") Long flowId); + + @Select("select is_admin from dss_user where username = #{userName}") + boolean isAdmin(@Param("userName") String userName); + + @Select({ + "" + }) + @Results({ + @Result(property = "creator", column = "created_by"), + @Result(property = "username", column = "username"), + @Result(property = "joinTime", column = "join_time"), + @Result(property = "workspaceId", column = "workspace_id") + }) + List getWorkspaceUsers(@Param("workspaceId") String workspaceId, + @Param("department") String department, + @Param("username") String username, @Param("roleId") int roleId); + + @Select("select * from dss_workspace_user where username in " + + "(select username from dss_workspace_user_role where role_id = #{roleId} and workspace_id = #{workspaceId})" + + "and workspace_id = #{workspaceId}") + @Results({ + @Result(property = "creator", column = "created_by"), + @Result(property = "username", column = "username"), + @Result(property = "joinTime", column = "join_time"), + @Result(property = "workspaceId", column = "workspace_id") + }) + List getWorkspaceUsersByRole(@Param("workspaceId") int workspaceId, @Param("roleId") int roleId); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/LinkisUserMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/LinkisUserMapper.java new file mode 100644 index 000000000..9b2f603fe --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/LinkisUserMapper.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.LinkisUser; +import org.apache.ibatis.annotations.Mapper; + + +@Mapper +public interface LinkisUserMapper extends BaseMapper { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/SidebarContentMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/SidebarContentMapper.java new file mode 100644 index 000000000..2043bf493 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/SidebarContentMapper.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.SidebarContent; +import org.apache.ibatis.annotations.Mapper; + + +@Mapper +public interface SidebarContentMapper extends BaseMapper { +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/SidebarMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/SidebarMapper.java new file mode 100644 index 000000000..7533055d5 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/SidebarMapper.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.Sidebar; +import org.apache.ibatis.annotations.Mapper; + + +@Mapper +public interface SidebarMapper extends BaseMapper { + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/WorkspaceMapper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/WorkspaceMapper.java new file mode 100644 index 000000000..9e4c05697 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/WorkspaceMapper.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.dao; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSFavorite; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspace; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.*; +import org.apache.ibatis.annotations.Param; + +import java.util.List; + + +public interface WorkspaceMapper { + + List getWorkspaces(); + + /** + * 获取空间名称 + * @param name + * @return + */ + List findByWorkspaceName(@Param("name") String name); + + void addWorkSpace(DSSWorkspace dssWorkspace); + + List getHomepageDemoMenusEn(); + List getHomepageDemoMenusCn(); + + List getHomepageInstancesByMenuIdCn(Long id); + List getHomepageInstancesByMenuIdEn(Long id); + + List getHomepageVideosEn(); + List getHomepageVideosCn(); + + DSSWorkspace getWorkspaceById(@Param("workspaceId") Long workspaceId); + + List getManagementMenuCn(); + List getManagementMenuEn(); + + List getApplicationMenuCn(); + List getApplicationMenuEn(); + + List getMenuAppInstancesCn(Long id); + List getMenuAppInstancesEn(Long id); + + List getWorkspaceFavoritesCn(@Param("username") String username, @Param("workspaceId") Long workspaceId); + + List getWorkspaceFavoritesEn(@Param("username") String username, @Param("workspaceId") Long workspaceId); + + void addFavorite(DSSFavorite dssFavorite); + + void deleteFavorite(Long favouritesId); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSComponentRoleMapper.xml b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSComponentRoleMapper.xml new file mode 100644 index 000000000..d3bcb4c77 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSComponentRoleMapper.xml @@ -0,0 +1,31 @@ + + + + + + + + insert into dss_component_role (workspace_id, component_id, role_id, + priv, update_time, updateby) + values + + (#{item.workspaceId},#{item.componentId},#{item.roleId}, + #{item.priv},#{item.updateTime},#{item.updateBy}) + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSMenuRoleMapper.xml b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSMenuRoleMapper.xml new file mode 100644 index 000000000..b66b89cc7 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSMenuRoleMapper.xml @@ -0,0 +1,31 @@ + + + + + + + + insert into dss_menu_role (workspace_id,menu_id,role_id,priv, update_time,updateby) + values + + ( #{item.workspaceId},#{item.menuId},#{item.roleId}, + #{item.priv},#{item.updateTime},#{item.updateBy}) + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceHomepageMapper.xml b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceHomepageMapper.xml new file mode 100644 index 000000000..6638c62df --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceHomepageMapper.xml @@ -0,0 +1,29 @@ + + + + + + + + insert into dss_workspace_homepage (workspace_id, role_id, homepage_url, update_time) + values + + #{homepage.workspaceId}, #{homepage.roleId}, #{homepage.homepageUrl}, #{homepage.updateTime} + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceMapper.xml b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceMapper.xml new file mode 100644 index 000000000..895f0d9eb --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceMapper.xml @@ -0,0 +1,81 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + id,`name`,`label`,`description`,`create_by`,`create_time`,`department`,`product`,`source`,`last_update_time`,`last_update_user` + + + + INSERT INTO dss_workspace() + VALUES + (#{id},#{name},#{label},#{description},#{createBy},#{createTime},#{department},#{product},#{source},#{lastUpdateTime},#{lastUpdateUser}) + + + + id,`workspace_id`,`username`,`join_time`,`created_by` + + + + + + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceUserMapper.xml b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceUserMapper.xml new file mode 100644 index 000000000..5640baf26 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/DSSWorkspaceUserMapper.xml @@ -0,0 +1,29 @@ + + + + + + + + + + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/WorkspaceMapper.xml b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/WorkspaceMapper.xml new file mode 100644 index 000000000..202314df7 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/dao/impl/WorkspaceMapper.xml @@ -0,0 +1,193 @@ + + + + + + + + id,`name`,`label`,`description`,`department`,`product`,`source`,`create_by`,`create_time`,`last_update_user`,`last_update_time` + + + + m.`id`,m.`title_cn` AS `title`, m.`desc_cn` AS `description`,m.`labels_cn` AS `labels`,m.image, + m.`access_button_cn` AS `access_button`,m.`manual_button_cn` AS `manualButton`,m.`is_active`, + m.`manual_button_url`,m.`icon`,m.`order`,app.`homepage_url` AS `access_button_url`,app.project_url, app.`name` + + + m.`id`,m.`title_en` AS `title`, m.`desc_en` AS `description`,m.`labels_en` AS `labels`,m.image, + m.`access_button_en` AS `access_button`,m.`manual_button_en` AS `manualButton`,m.`is_active`, + m.`manual_button_url`,m.`icon`,m.`order`,app.`homepage_url` AS `access_button_url`,app.project_url, app.`name` + + + + id,`username`,`workspace_id`,`menu_application_id`,`order`,`create_by`,`create_time`,`last_update_user`,`last_update_time` + + + + + + + + + + INSERT INTO dss_workspace () + VALUES + (#{id},#{name},#{label},#{description},#{department},#{product},#{source},#{createBy},#{createTime},#{lastUpdateUser},#{lastUpdateTime}) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + INSERT INTO dss_onestop_user_favorites () + VALUES + (#{id},#{username},#{workspaceId},#{menuApplicationId},#{order},#{createBy},#{createTime},#{lastUpdateUser},#{lastUpdateTime}) + + + + DELETE + FROM + dss_onestop_user_favorites + WHERE + id = #{favouritesId} + + \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/exception/DSSWorkspaceDuplicateNameException.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/exception/DSSWorkspaceDuplicateNameException.java new file mode 100644 index 000000000..1d7e8900b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/exception/DSSWorkspaceDuplicateNameException.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class DSSWorkspaceDuplicateNameException extends ErrorException { + + public DSSWorkspaceDuplicateNameException(int errCode, String desc) { + super(errCode, desc); + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/exception/DSSWorkspaceLoginFailException.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/exception/DSSWorkspaceLoginFailException.java new file mode 100644 index 000000000..0fb0359be --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/exception/DSSWorkspaceLoginFailException.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class DSSWorkspaceLoginFailException extends ErrorException { + public DSSWorkspaceLoginFailException(int errCode, String desc) { + super(errCode, desc); + } + + public DSSWorkspaceLoginFailException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSDictionaryRestful.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSDictionaryRestful.java new file mode 100644 index 000000000..a3ce4d764 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSDictionaryRestful.java @@ -0,0 +1,134 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.restful; + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSDictionary; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSDictionaryRequestVO; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSDictionaryService; +import com.webank.wedatasphere.dss.framework.workspace.util.RestfulUtils; +import com.webank.wedatasphere.linkis.server.Message; +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang.StringUtils; +import org.apache.commons.math3.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; +import org.springframework.web.bind.annotation.RequestBody; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.Consumes; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.List; +import java.util.Map; + + +@Component +@Path("/dss/framework/workspace/") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSDictionaryRestful { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSDictionaryRestful.class); + + @Autowired + private DSSDictionaryService dictionaryService; + + /** + * 数据字典 - 根据key获取 + * @param request + * @param dictionaryRequestVO + * @return + */ + @POST + @Path("getDicList") + public Response getDevFlowList(@Context HttpServletRequest request, @RequestBody DSSDictionaryRequestVO dictionaryRequestVO){ + try{ + if(dictionaryRequestVO.getWorkspaceId()==null){ + return Message.messageToResponse(Message.error("workspaceId(空间id)不能为空")); + } + if(StringUtils.isEmpty(dictionaryRequestVO.getParentKey())&&StringUtils.isEmpty(dictionaryRequestVO.getDicKey())){ + return Message.messageToResponse(Message.error("获取的parentKey和dicKey不能同时为空")); + } + List dictionaries = dictionaryService.getListByParam(dictionaryRequestVO); + boolean isEnglish = "en".equals(request.getHeader("Content-language")); + if(CollectionUtils.isNotEmpty(dictionaries) && isEnglish){ + dictionaries.stream().forEach(e->international(e)); + } + return RestfulUtils.dealOk("获取数据字典成功", new Pair<>("list", dictionaries)); + }catch(Exception e){ + LOGGER.error("Fail to get getDevFlowListError for user {} in workspace {}",dictionaryRequestVO.getWorkspaceId(), e); + return RestfulUtils.dealError("获取数据字典失败:"+e.getMessage()); + } + } + + @POST + @Path("getDicSecondList") + public Response getDicSecondList(@Context HttpServletRequest request, @RequestBody DSSDictionaryRequestVO dictionaryRequestVO){ + try{ + if(dictionaryRequestVO.getWorkspaceId()==null){ + return Message.messageToResponse(Message.error("workspaceId(空间id)不能为空")); + } + if(StringUtils.isEmpty(dictionaryRequestVO.getParentKey())&&StringUtils.isEmpty(dictionaryRequestVO.getDicKey())){ + return Message.messageToResponse(Message.error("获取的parentKey和dicKey不能同时为空")); + } + Map map = dictionaryService.getDicSecondList(dictionaryRequestVO); + boolean isEnglish = "en".equals(request.getHeader("Content-language")); + if(map!=null && map.size() > 0 && isEnglish){ + List dictionaries = (List) map.get("list"); + if(CollectionUtils.isNotEmpty(dictionaries)){ + dictionaries.stream().forEach(e->international(e)); + } + Map> mapList = ( Map>) map.get("mapList"); + if(mapList!=null && mapList.size() > 0){ + for(String key : mapList.keySet()){ + mapList.get(key).stream().forEach(e->international(e)); + } + } + } + return RestfulUtils.dealOk("获取数据字典成功", new Pair<>("list", map)); + }catch(Exception e){ + LOGGER.error("Fail to get getDicSecondListError for user {} in workspace {}",dictionaryRequestVO.getWorkspaceId(), e); + return RestfulUtils.dealError("获取数据字典失败:"+e.getMessage()); + } + } + + + //国际化，由于前端只是使用dicName，所以在英文的时候直接将dicNameEn赋值给dicName即可 + public void international(DSSDictionary dssDictionary){ + if(dssDictionary==null){ + return; + } + if(StringUtils.isNotBlank(dssDictionary.getDicNameEn())){ + dssDictionary.setDicName(dssDictionary.getDicNameEn()); + dssDictionary.setDicNameEn(null); + } + if(StringUtils.isNotBlank(dssDictionary.getDicValueEn())){ + dssDictionary.setDicValue(dssDictionary.getDicValueEn()); + dssDictionary.setDicValueEn(null); + } + if(StringUtils.isNotBlank(dssDictionary.getTitleEn())){ + dssDictionary.setTitle(dssDictionary.getTitleEn()); + dssDictionary.setTitleEn(null); + } + } +} \ No newline at end of file diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSSideInfoRestful.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSSideInfoRestful.java new file mode 100644 index 000000000..259707f81 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSSideInfoRestful.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.restful; + +import com.webank.wedatasphere.dss.framework.workspace.service.DSSSideInfoService; +import com.webank.wedatasphere.dss.framework.workspace.util.RestfulUtils; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.math3.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; + + +@Component +@Path("/dss/framework/workspace/") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSSideInfoRestful { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSSideInfoRestful.class); + + @Autowired + private DSSSideInfoService dssSideInfoService; + + @GET + @Path("getSideInfos") + public Response getSideInfos(@Context HttpServletRequest request, @QueryParam("workspaceId") Integer workspaceId){ + String username = SecurityFilter.getLoginUsername(request); + try{ + boolean isEnglish = "en".equals(request.getHeader("Content-language")); + return RestfulUtils.dealOk("获取侧边栏成功", new Pair<>("presentations", dssSideInfoService.getSidebarVOList(username, workspaceId,isEnglish))); + }catch(Exception e){ + LOGGER.info("Fail to get sideinfos for user {} in workspace {}",username, workspaceId, e); + return RestfulUtils.dealError("获取侧边栏失败"); + } + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspacePrivRestful.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspacePrivRestful.java new file mode 100644 index 000000000..bc96526fc --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspacePrivRestful.java @@ -0,0 +1,122 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.restful; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceHomepageSettingVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspacePrivVO; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspacePrivService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceService; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceDBHelper; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.math3.util.Pair; +import org.codehaus.jackson.JsonNode; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.ArrayList; +import java.util.List; + +import static com.webank.wedatasphere.dss.framework.workspace.util.DSSWorkspaceConstant.WORKSPACE_ID_STR; + + +@Component +@Path("/dss/framework/workspace") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSWorkspacePrivRestful { + + @Autowired + DSSWorkspaceService dssWorkspaceService; + @Autowired + DSSWorkspacePrivService dssWorkspacePrivService; + @Autowired + WorkspaceDBHelper workspaceDBHelper; + + @GET + @Path("getWorkspaceMenuPrivs") + public Response getWorkspaceMenuPrivs(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) String workspaceId){ + //todo 返回工作空间中角色对菜单的访问权限 + DSSWorkspacePrivVO workspaceMenuPrivs = dssWorkspaceService.getWorkspaceMenuPrivs(workspaceId); + return Message.messageToResponse(Message.ok().data("workspaceMenuPrivs", workspaceMenuPrivs)); + } + + @GET + @Path("getWorkspaceComponentPrivs") + public Response getWorkspaceComponentPrivs(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) String workspaceId){ + return Message.messageToResponse(Message.ok("接口废弃")); + } + + @GET + @Path("getWorkspaceHomepageSettings") + public Response getWorkspaceHomepageSettings(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) int workspaceId){ + String username = SecurityFilter.getLoginUsername(request); + DSSWorkspaceHomepageSettingVO dssWorkspaceHomepageSettingVO = dssWorkspaceService.getWorkspaceHomepageSettings(workspaceId); + return Message.messageToResponse(Message.ok().data("homepageSettings", dssWorkspaceHomepageSettingVO)); + } + + @POST + @Path("updateRoleMenuPriv") + public Response updateRoleMenuPriv(@Context HttpServletRequest request, JsonNode jsonNode){ + String updater = SecurityFilter.getLoginUsername(request); + int menuId = jsonNode.get("menuId").getIntValue(); + int workspaceId = jsonNode.get("workspaceId").getIntValue(); + JsonNode menuPrivs = jsonNode.get("menuPrivs"); + List> pairs = new ArrayList<>(); + menuPrivs.getFields().forEachRemaining(field -> { + Integer roleId = dssWorkspacePrivService.getRoleId(workspaceId, field.getKey()); + if (roleId == null) { + roleId = workspaceDBHelper.getRoleIdByName(field.getKey()); + } + pairs.add(new Pair(roleId, field.getValue().getBooleanValue())); + }); + dssWorkspacePrivService.updateRoleMenuPriv(workspaceId, menuId, updater, pairs); + return Message.messageToResponse(Message.ok("更新角色对于菜单的权限成功")); + } + + @POST + @Path("updateRoleComponentPriv") + public Response updateRoleComponentPriv(@Context HttpServletRequest request, JsonNode jsonNode){ + //todo 更新工作空间中角色对于component的权限 + String username = SecurityFilter.getLoginUsername(request); + int menuId = jsonNode.get("componentId").getIntValue(); + int workspaceId = jsonNode.get("workspaceId").getIntValue(); + JsonNode componentPrivs = jsonNode.get("componentPrivs"); + List> pairs = new ArrayList<>(); + componentPrivs.getFields().forEachRemaining(field -> { + Integer roleId = dssWorkspacePrivService.getRoleId(workspaceId, field.getKey()); + if (roleId == null) { + roleId = workspaceDBHelper.getRoleIdByName(field.getKey()); + } + pairs.add(new Pair(roleId, field.getValue().getBooleanValue())); + }); + dssWorkspacePrivService.updateRoleComponentPriv(workspaceId, menuId, username, pairs); + return Message.messageToResponse(Message.ok().data("updateRoleComponentPriv","更新组件权限成功")); + } + + @POST + @Path("updateRoleHomepage") + public Response updateRoleHomepage(@Context HttpServletRequest request, JsonNode jsonNode){ + return null; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceRestful.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceRestful.java new file mode 100644 index 000000000..b931eafb2 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceRestful.java @@ -0,0 +1,134 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.restful; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspace; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceHomePageVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceOverviewVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DepartmentVO; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceMenuService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceService; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceDBHelper; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.codehaus.jackson.JsonNode; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.ArrayList; +import java.util.List; + +import static com.webank.wedatasphere.dss.framework.workspace.util.DSSWorkspaceConstant.WORKSPACE_ID_STR; + + + + +@Component +@Path("/dss/framework/workspace") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSWorkspaceRestful { + @Autowired + private DSSWorkspaceService dssWorkspaceService; + @Autowired + private DSSWorkspaceMenuService dssWorkspaceMenuService; + @Autowired + private WorkspaceDBHelper workspaceDBHelper; + + @POST + @Path("/createWorkspace") + public Response createWorkspace(@Context HttpServletRequest request, JsonNode node)throws ErrorException { + String userName = SecurityFilter.getLoginUsername(request); + if (!dssWorkspaceService.checkAdmin(userName)){ + return Message.messageToResponse(Message.error("您好，您不是管理员,没有权限建立工作空间")); + } + String workSpaceName = node.get("workspaceName").getTextValue(); + String department = node.get("department").getTextValue(); + String description = node.get("description").getTextValue(); + String stringTags = node.get("tags").getTextValue(); + String productName = node.get("productName").getTextValue(); + int workspaceId = dssWorkspaceService.createWorkspace(workSpaceName, stringTags, userName, description, department, productName); + return Message.messageToResponse(Message.ok().data("workspaceId", workspaceId).data("workspaceName",workSpaceName)); + } + + @GET + @Path("listDepartments") + public Response listDepartments(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) String workspaceId){ + //todo 要从um中获取 + List departments = dssWorkspaceService.getDepartments(); + return Message.messageToResponse(Message.ok().data("departments", departments)); + } + + @GET + @Path("getWorkspaces") + public Response getWorkspaces(@Context HttpServletRequest request){ + String username = SecurityFilter.getLoginUsername(request); + List workspaces = dssWorkspaceService.getWorkspaces(username); + List dssWorkspaceVOS = new ArrayList<>(); + for (DSSWorkspace workspace:workspaces ){ + String name = workspace.getName(); + int id = workspace.getId(); + String labels = workspace.getLabel(); + DSSWorkspaceVO dssWorkspaceVO = new DSSWorkspaceVO(); + dssWorkspaceVO.setId(id); + dssWorkspaceVO.setName(name); + dssWorkspaceVO.setTags(labels); + dssWorkspaceVO.setDepartment(workspace.getDepartment()); + dssWorkspaceVO.setDescription(workspace.getDescription()); + dssWorkspaceVO.setProduct(workspace.getProduct()); + dssWorkspaceVOS.add(dssWorkspaceVO); + } + //todo 获取用户所有所有能够访问的工作空间 + return Message.messageToResponse(Message.ok().data("workspaces", dssWorkspaceVOS)); + } + + @GET + @Path("getWorkspaceHomePage") + public Response getWorkspaceHomePage(@Context HttpServletRequest request, @QueryParam("micro_module") String moduleName){ + //如果用户的工作空间大于两个，那么就直接返回/workspace页面 + String username = SecurityFilter.getLoginUsername(request); + DSSWorkspaceHomePageVO dssWorkspaceHomePageVO = dssWorkspaceService.getWorkspaceHomePage(username,moduleName); + return Message.messageToResponse(Message.ok().data("workspaceHomePage", dssWorkspaceHomePageVO)); + } + + @GET + @Path("getOverview") + public Response getOverview(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) int workspaceId){ + String username = SecurityFilter.getLoginUsername(request); + String language = request.getHeader("Content-language"); + boolean isEnglish = "en".equals(language); + DSSWorkspaceOverviewVO dssWorkspaceOverviewVO = dssWorkspaceService.getOverview(username, workspaceId, isEnglish); + return Message.messageToResponse(Message.ok().data("overview", dssWorkspaceOverviewVO)); + } + + @GET + @Path("refreshCache") + public Response refreshCache(@Context HttpServletRequest request){ + workspaceDBHelper.retrieveFromDB(); + return Message.messageToResponse(Message.ok("refresh ok")); + } + +} + diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceRoleRestful.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceRoleRestful.java new file mode 100644 index 000000000..62c046f98 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceRoleRestful.java @@ -0,0 +1,212 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.restful; + + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSApplication; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceRoleVO; +import com.webank.wedatasphere.dss.framework.workspace.constant.ApplicationConf; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSApplicationMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSUserService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceRoleService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceService; +import com.webank.wedatasphere.dss.framework.workspace.util.ApplicationUtils; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.lang.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.Cookie; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; + + +@Component +@Path("/dss/framework/workspace") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSWorkspaceRoleRestful { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSWorkspaceRoleRestful.class); + private static final String WORKSPACE_ID_STR = "workspaceId"; + private static final String DEFAULT_PATH = "/api/rest_j/v1"; + @Autowired + DSSWorkspaceService dssWorkspaceService; + @Autowired + private DSSWorkspaceRoleService dssWorkspaceRoleService; + @Autowired + private DSSUserService userService; + @Autowired + private DSSApplicationMapper applicationMapper; + + @GET + @Path("getBaseInfo") + public Response getBaseInfo(@Context HttpServletRequest req) { + String username = SecurityFilter.getLoginUsername(req); + userService.saveWorkspaceUser(username); + List applicationList = applicationMapper.selectList(null); + DSSApplication schedulis = new DSSApplication(); + schedulis.setName(ApplicationConf.SCHEDULER_APP_CONN_NAME); + String url = ApplicationConf.SCHEDULIS_URL.getValue(); + schedulis.setUrl(url); + schedulis.setHomepageUrl(url + "/homepage"); + schedulis.setProjectUrl(url + "/manager?project=${projectName}"); + schedulis.setRedirectUrl(url + "/api/v1/redirect"); + schedulis.setIfIframe(true); + applicationList.add(schedulis); + for (DSSApplication application : applicationList) { + String redirectUrl = application.getRedirectUrl(); + String enhanceJson = application.getEnhanceJson(); + if (redirectUrl != null) { + application.setHomepageUrl(ApplicationUtils.redirectUrlFormat(redirectUrl, application.getHomepageUrl())); + application.setProjectUrl(ApplicationUtils.redirectUrlFormat(redirectUrl, application.getProjectUrl())); + if (StringUtils.isNotEmpty(enhanceJson) && enhanceJson.contains("scheduleHistory")) { + Gson gson = new Gson(); + HashMap scheduleHistoryMap = gson.fromJson(enhanceJson, HashMap.class); + String oldUrl = scheduleHistoryMap.get("scheduleHistory").toString(); + String formatUrl = ApplicationUtils.redirectUrlFormat(redirectUrl, oldUrl); + scheduleHistoryMap.replace("scheduleHistory", formatUrl); + application.setEnhanceJson(gson.toJson(scheduleHistoryMap)); + } + } + } + //返回FAQ地址 + String faqUrl = ApplicationConf.FAQ.getValue(); + boolean isAdmin = userService.isAdminUser(username); + //前台需要返回username + return Message.messageToResponse(Message.ok().data("applications", applicationList). + data("username", username).data("isAdmin", isAdmin) + .data("DWSParams", Collections.singletonMap("faq", faqUrl))); + } + + @GET + @Path("getWorkspaceRoles") + public Response getWorkspaceRoles(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) int workspaceId){ + //todo 获取工作空间中所有的角色 + List workspaceRoles = dssWorkspaceService.getWorkspaceRoles(workspaceId); + return Message.messageToResponse(Message.ok().data("workspaceRoles", workspaceRoles)); + } + + + @POST + @Path("addWorkspaceRole") + public Response addWorkspaceRole(@Context HttpServletRequest request, JsonNode jsonNode){ + String username = SecurityFilter.getLoginUsername(request); + int workspaceId = jsonNode.get("workspaceId").getIntValue(); + String roleName = jsonNode.get("roleName").getTextValue(); + List menuIds = new ArrayList<>(); + List componentIds = new ArrayList<>(); + JsonNode menuIdsNode = jsonNode.get("menuIds"); + if (menuIdsNode != null && menuIdsNode.isArray()){ + menuIdsNode.forEach(tmpMenuId -> menuIds.add(tmpMenuId.getIntValue())); + } + JsonNode componentIdsNode = jsonNode.get("componentIds"); + if (componentIdsNode != null && componentIdsNode.isArray()){ + componentIdsNode.forEach(tmpMenuId -> componentIds.add(tmpMenuId.getIntValue())); + } + dssWorkspaceRoleService.addWorkspaceRole(roleName, workspaceId, menuIds, componentIds, username); + return Message.messageToResponse(Message.ok("创建角色成功")); + } + + @POST + @Path("deleteWorkspaceRole") + public Response deleteWorkspaceRole(@Context HttpServletRequest request, JsonNode jsonNode){ + + return null; + } + + @GET + @Path("getWorkspaceBaseInfo") + public Response getWorkspaceInfo(@Context HttpServletRequest request, + @Context HttpServletResponse response, + @QueryParam(WORKSPACE_ID_STR) Integer workspaceId){ + String username = SecurityFilter.getLoginUsername(request); + //如果workspaceId为null的话,那么就找到这个用户工作空间 + if (workspaceId == null || workspaceId <= 0){ + workspaceId = dssWorkspaceRoleService.getWorkspaceIdByUser(username); + } + //将workspaceId作为cookie写入 + Cookie[] cookies = request.getCookies(); + for(Cookie cookie : cookies){ + if(WORKSPACE_ID_STR.equals(cookie.getName())){ + cookie.setMaxAge(0); + cookie.setPath("/"); + cookie.setValue(null); + response.addCookie(cookie); + break; + } + } + Cookie workspaceCookie = new Cookie(WORKSPACE_ID_STR, workspaceId.toString()); + workspaceCookie.setPath("/"); + response.addCookie(workspaceCookie); + List roles = dssWorkspaceRoleService.getRoleInWorkspace(username, workspaceId); + if(roles == null || roles.isEmpty()){ + LOGGER.error("username {}, in workspace {} roles are null or empty", username, workspaceId); + return Message.messageToResponse(Message.error("can not get roles information")); + } + //判断如果是没有权限的，那么就直接干掉 + if (roles.contains("apiUser")){ + int priv = dssWorkspaceRoleService.getApiPriv(username, workspaceId, "apiUser", "apiService"); + if(priv <= 0) { + roles.remove("apiUser"); + } + } + Message retMessage = Message.ok(); + //工作空间中，加上用户在顶部的菜单 + if (roles.contains("analyser")){ + retMessage.data("topName", "Scriptis"); + retMessage.data("topUrl", "/home"); + } else if (roles.contains("developer")){ + retMessage.data("topName", "Scriptis"); + retMessage.data("topUrl", "/home"); + }else if(roles.contains("apiUser") && roles.size() == 1){ + retMessage.data("topName","Scriptis"); + retMessage.data("topUrl", "/home"); + }else{ + retMessage.data("topName", "Scriptis"); + retMessage.data("topUrl", "/home"); + } + //如果其他的角色也是有这个api权限的，那么就加上这个apiUser + boolean flag = false; + for (String role : roles){ + int priv = dssWorkspaceRoleService.getApiPriv(username, workspaceId, role, "apiService"); + if (priv >= 1) { + flag = true; + break; + } + } + if(flag && !roles.contains("apiUser")){ + roles.add("apiUser"); + } + //roles.add("apiUser"); + return Message.messageToResponse(retMessage.data("roles", roles).data("workspaceId", workspaceId)); + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceUserRestful.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceUserRestful.java new file mode 100644 index 000000000..46b063216 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/DSSWorkspaceUserRestful.java @@ -0,0 +1,169 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.restful; + +import com.github.pagehelper.PageInfo; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSUser; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceRoleVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceUserVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.StaffInfoVO; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceMenuService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceUserService; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceDBHelper; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.apache.commons.lang.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +import static com.webank.wedatasphere.dss.framework.workspace.util.DSSWorkspaceConstant.WORKSPACE_ID_STR; + + + +@Component +@Path("/dss/framework/workspace") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class DSSWorkspaceUserRestful { + + @Autowired + private DSSWorkspaceService dssWorkspaceService; + @Autowired + private DSSWorkspaceMenuService dssWorkspaceMenuService; + @Autowired + private WorkspaceDBHelper workspaceDBHelper; + @Autowired + private DSSWorkspaceUserService dssWorkspaceUserService; + + @GET + @Path("getWorkspaceUsers") + public Response getWorkspaceUsers(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) String workspaceId, + @QueryParam("pageNow") Integer pageNow, @QueryParam("pageSize") Integer pageSize, + @QueryParam("department") String department, @QueryParam("username") String username, + @QueryParam("roleName") String roleName){ + //todo 获取工作空间中所有的用户以及他们的角色信息 + if(pageNow == null){ + pageNow = 1; + } + if(pageSize == null){ + //默认改成20 + pageSize = 20; + } + if(StringUtils.isNotEmpty(roleName)){ + //如果roleName不是空的话，就按照roleName来吧 + List totals = new ArrayList<>(); + List workspaceUsers = + dssWorkspaceService.getWorkspaceUsersByRole(Integer.parseInt(workspaceId),roleName, totals, pageNow, pageSize); + PageInfo pageInfo = new PageInfo<>(workspaceUsers); + List list = pageInfo.getList(); + long total = pageInfo.getTotal(); + List dssRoles = workspaceDBHelper.getRoleVOs(Integer.parseInt(workspaceId)); + return Message.messageToResponse(Message.ok().data("roles", dssRoles).data("workspaceUsers", list).data("total", totals.get(0))); + }else{ + List totals = new ArrayList<>(); + List workspaceUsers = + dssWorkspaceService.getWorkspaceUsers(workspaceId, department, username, roleName, pageNow, pageSize, totals); + PageInfo pageInfo = new PageInfo<>(workspaceUsers); + List list = pageInfo.getList(); + long total = pageInfo.getTotal(); + List dssRoles = workspaceDBHelper.getRoleVOs(Integer.parseInt(workspaceId)); + return Message.messageToResponse(Message.ok().data("roles", dssRoles).data("workspaceUsers", list).data("total", totals.get(0))); + } + } + + @GET + @Path("getAllWorkspaceUsers") + public Response getAllWorkspaceUsers(@Context HttpServletRequest request, @QueryParam(WORKSPACE_ID_STR) int workspaceId ){ + String username = SecurityFilter.getLoginUsername(request); + List users = dssWorkspaceUserService.getAllWorkspaceUsers(workspaceId); + return Message.messageToResponse(Message.ok().data("users", users)); + } + + @POST + @Path("addWorkspaceUser") + public Response addWorkspaceUser(@Context HttpServletRequest request, JsonNode jsonNode){ + //todo 工作空间添加用户 + String creator = SecurityFilter.getLoginUsername(request); + List roles = new ArrayList<>(); + if (jsonNode.get("roles").isArray()){ + for (JsonNode role : jsonNode.get("roles")){ + roles.add(role.getIntValue()); + } + } + int workspaceId = jsonNode.get("workspaceId").getIntValue(); + String userName = jsonNode.get("username").getTextValue(); + dssWorkspaceService.addWorkspaceUser(roles, workspaceId, userName, creator); + return Message.messageToResponse(Message.ok()); + } + + @POST + @Path("updateWorkspaceUser") + public Response updateWorkspaceUser(@Context HttpServletRequest request, JsonNode jsonNode){ + String creator = SecurityFilter.getLoginUsername(request); + List roles = new ArrayList<>(); + if (jsonNode.get("roles").isArray()){ + for (JsonNode role : jsonNode.get("roles")){ + roles.add(role.getIntValue()); + } + } + int workspaceId = jsonNode.get("workspaceId").getIntValue(); + String userName = jsonNode.get("username").getTextValue(); + dssWorkspaceUserService.updateWorkspaceUser(roles, workspaceId, userName, creator); + return Message.messageToResponse(Message.ok()); + } + + @POST + @Path("deleteWorkspaceUser") + public Response deleteWorkspaceUser(@Context HttpServletRequest request, JsonNode jsonNode){ + //todo 删除工作空间中的用户 + String userName = jsonNode.get("username").getTextValue(); + int workspaceId = jsonNode.get("workspaceId").getIntValue(); + dssWorkspaceUserService.deleteWorkspaceUser(userName,workspaceId); + return Message.messageToResponse(Message.ok()); + } + + @GET + @Path("listAllUsers") + public Response listAllUsers(@Context HttpServletRequest request){ + List dssUsers = dssWorkspaceUserService.listAllDSSUsers(); + return Message.messageToResponse(Message.ok().data("users", dssUsers)); + } + + @GET + @Path("getWorkspaceIdByUserName") + public Response getWorkspaceIdByUserName(@Context HttpServletRequest request,@QueryParam("userName") String userName){ + String loginUserName = SecurityFilter.getLoginUsername(request); + String queryUserName = userName; + if(StringUtils.isEmpty(userName)){ + queryUserName = loginUserName; + } + List userWorkspaceIds = dssWorkspaceUserService.getUserWorkspaceIds(queryUserName); + String userWorkspaceIdStr = userWorkspaceIds.stream().map(x->x.toString()).collect(Collectors.joining(",")); + return Message.messageToResponse(Message.ok().data("userWorkspaceIds", userWorkspaceIdStr)); + } +} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/WorkspaceRestfulApi.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/WorkspaceRestfulApi.java similarity index 78% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/WorkspaceRestfulApi.java rename to dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/WorkspaceRestfulApi.java index 4f85a55a4..95ab2093a 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/WorkspaceRestfulApi.java +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/restful/WorkspaceRestfulApi.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,19 +14,21 @@ * */ -package com.webank.wedatasphere.dss.server.restful; +package com.webank.wedatasphere.dss.framework.workspace.restful; -import com.webank.wedatasphere.dss.server.dto.response.HomepageDemoMenuVo; -import com.webank.wedatasphere.dss.server.dto.response.HomepageVideoVo; -import com.webank.wedatasphere.dss.server.dto.response.OnestopMenuVo; -import com.webank.wedatasphere.dss.server.entity.DSSWorkspace; -import com.webank.wedatasphere.dss.server.dto.response.WorkspaceDepartmentVo; -import com.webank.wedatasphere.dss.server.dto.response.*; -import com.webank.wedatasphere.dss.server.service.DSSUserService; -import com.webank.wedatasphere.dss.server.service.DSSWorkspaceService; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspace; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.HomepageDemoMenuVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.HomepageVideoVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.OnestopMenuVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.WorkspaceFavoriteVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DepartmentVO; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceService; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; import com.webank.wedatasphere.linkis.server.Message; import com.webank.wedatasphere.linkis.server.security.SecurityFilter; import org.codehaus.jackson.JsonNode; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; @@ -36,28 +37,26 @@ import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; +import java.util.HashSet; import java.util.List; +import java.util.Set; + -/** - * Created by schumiyi on 2020/6/19 - */ @Component -@Path("/dss") +@Path("/dss/framework/workspace") @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.APPLICATION_JSON) public class WorkspaceRestfulApi { - + private static final Logger LOGGER = LoggerFactory.getLogger(WorkspaceRestfulApi.class); @Autowired private DSSWorkspaceService dssWorkspaceService; - @Autowired - private DSSUserService dssUserService; - @GET @Path("/workspaces") public Response getAllWorkspaces(@Context HttpServletRequest req) { // TODO: Order By time - List workspaces = dssWorkspaceService.getWorkspaces(); + String username = SecurityFilter.getLoginUsername(req); + List workspaces = dssWorkspaceService.getWorkspaces(username); return Message.messageToResponse(Message.ok().data("workspaces", workspaces)); } @@ -71,7 +70,7 @@ public Response getWorkspacesById(@Context HttpServletRequest req, @PathParam("i @GET @Path("/workspaces/departments") public Response getAllWorkspaceDepartments(@Context HttpServletRequest req) { - List departments = dssWorkspaceService.getWorkSpaceDepartments(); + List departments = dssWorkspaceService.getDepartments(); return Message.messageToResponse(Message.ok().data("departments", departments)); } @@ -84,8 +83,11 @@ public Response getUsernameExistence(@Context HttpServletRequest req, @QueryPara @POST @Path("/workspaces") - public Response addWorkspace(@Context HttpServletRequest req, JsonNode json) { + public Response addWorkspace(@Context HttpServletRequest req, JsonNode json) throws ErrorException { String userName = SecurityFilter.getLoginUsername(req); + if (!dssWorkspaceService.checkAdmin(userName)){ + return Message.messageToResponse(Message.error("您好，您不是管理员,没有权限建立工作空间")); + } String name = json.get("name").getTextValue(); if (dssWorkspaceService.existWorkspaceName(name)) { return Message.messageToResponse(Message.error("工作空间名重复")); @@ -93,7 +95,8 @@ public Response addWorkspace(@Context HttpServletRequest req, JsonNode json) { String department = json.get("department").getTextValue(); String label = json.get("label").getTextValue(); String description = json.get("description").getTextValue(); - Long workspaceId = dssWorkspaceService.addWorkspace(userName, name, department, label, description); + String productName = "DSS"; + int workspaceId = dssWorkspaceService.createWorkspace(name, label, userName, description, department, productName); return Message.messageToResponse(Message.ok().data("workspaceId", workspaceId)); } @@ -121,7 +124,6 @@ public Response getWorkspaceManagements(@Context HttpServletRequest req, @PathPa String header = req.getHeader("Content-language").trim(); boolean isChinese = "zh-CN".equals(header); String username = SecurityFilter.getLoginUsername(req); - List managements = dssWorkspaceService.getWorkspaceManagements(workspaceId, username, isChinese); return Message.messageToResponse(Message.ok().data("managements", managements)); } @@ -143,7 +145,8 @@ public Response getWorkspaceFavorites(@Context HttpServletRequest req, @PathPara boolean isChinese = "zh-CN".equals(header); String username = SecurityFilter.getLoginUsername(req); List favorites = dssWorkspaceService.getWorkspaceFavorites(workspaceId, username, isChinese); - return Message.messageToResponse(Message.ok().data("favorites", favorites)); + Set favoriteVos = new HashSet<>(favorites); + return Message.messageToResponse(Message.ok().data("favorites", favoriteVos)); } /** @@ -165,8 +168,10 @@ public Response addFavorite(@Context HttpServletRequest req, @PathParam("workspa @DELETE @Path("/workspaces/{workspaceId}/favorites/{favouritesId}") public Response deleteFavorite(@Context HttpServletRequest req, @PathParam("workspaceId") Long workspaceId, @PathParam("favouritesId") Long favouritesId) { + LOGGER.info("deleteFavorite--workspaceId={}--favouritesId={}",workspaceId,favouritesId); String username = SecurityFilter.getLoginUsername(req); Long favoriteId = dssWorkspaceService.deleteFavorite(username, favouritesId); return Message.messageToResponse(Message.ok().data("favoriteId", favoriteId)); } + } diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSDictionaryService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSDictionaryService.java new file mode 100644 index 000000000..ab71a8185 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSDictionaryService.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSDictionary; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSDictionaryRequestVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DepartmentVO; + +import java.util.List; +import java.util.Map; + + +public interface DSSDictionaryService { + + public List getListByParam(DSSDictionaryRequestVO dictionaryRequestVO) ; + + /** + * 获取编排模式 + * @param dictionaryRequestVO + * @return + */ + public Map getDicSecondList(DSSDictionaryRequestVO dictionaryRequestVO) ; + + /** + * 获取空间默认部门 + * @return + */ + public List getDefaultDepartmentVOList(); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSSideInfoService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSSideInfoService.java new file mode 100644 index 000000000..230e914dd --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSSideInfoService.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.SidebarVO; + +import java.util.List; + + +public interface DSSSideInfoService { + + List getSidebarVOList(String username, Integer workspaceId,boolean isEnglish); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSUserService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSUserService.java new file mode 100644 index 000000000..f63f95ac1 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSUserService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSUser; + +public interface DSSUserService { + + public void saveWorkspaceUser(String userName); + + + public Long getUserID(String userName); + + + public DSSUser getByUsername(String userName); + + public boolean isAdminUser(String username); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceMenuService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceMenuService.java new file mode 100644 index 000000000..777bfc64b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceMenuService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSMenu; + +import java.util.List; + + +public interface DSSWorkspaceMenuService { + + + List getRealComponents(List subMenus, String workspaceId, String username); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspacePrivService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspacePrivService.java new file mode 100644 index 000000000..c964d0e50 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspacePrivService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + +import org.apache.commons.math3.util.Pair; + +import java.util.List; + + +public interface DSSWorkspacePrivService { + void updateRoleMenuPriv(int workspaceId, int menuId, String updater, List> pairs); + + void updateRoleComponentPriv(int workspaceId, int menuId, String username, List> pairs); + + Integer getRoleId(int workspaceId, String key); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceRoleService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceRoleService.java new file mode 100644 index 000000000..c8de173f9 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceRoleService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + +import java.util.List; + + +public interface DSSWorkspaceRoleService { + void addWorkspaceRole(String roleName, int workspaceId, List menuIds, List componentIds, String username); + + List getRoleInWorkspace(String username, int workspaceId); + + Integer getWorkspaceIdByUser(String username); + + int getApiPriv(String username, Integer workspaceId, String roleName, String appName); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceService.java new file mode 100644 index 000000000..87d461b42 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceService.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSMenu; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspace; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.*; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.*; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + +import java.util.List; + + +public interface DSSWorkspaceService { + + + int createWorkspace(String workspaceName, String tags, String userName, String description, String department, String productName) throws ErrorException; + + void addWorkspaceUser(List roleIds, int workspaceId, String userName, String creater); + + List getWorkspaces(String userName); + + DSSWorkspaceHomePageVO getWorkspaceHomePage(String userName,String moduleName); + + List getWorkspaceMenus(String userName, String workspaceId); + + List getWorkspaceUsers(String workspaceId, String department, String username, + String roleName, int pageNow, int pageSize, List total); + + List getWorkspaceRoles(int workspaceId); + + DSSWorkspacePrivVO getWorkspaceMenuPrivs(String workspaceId); + + DSSWorkspaceOverviewVO getOverview(String username, int workspaceId, boolean isEnglish); + + DSSWorkspaceHomepageSettingVO getWorkspaceHomepageSettings(int workspaceId); + + String getWorkspaceName(String workspaceId); + + boolean checkAdmin(String userName); + + List getDepartments(); + + List getWorkspaceUsersByRole(int workspaceId, String roleName, List totals, + int pageNow, int pageSize); + + List getWorkspaces(); + + Long addWorkspace(String userName, String name, String department, String label, String description); + + boolean existWorkspaceName(String name); + + List getWorkSpaceDepartments(); + + List getHomepageDemos(boolean isChinese); + + List getHomepageVideos(boolean isChinese); + + List getWorkspaceManagements(Long workspaceId, String username, boolean isChinese); + + List getWorkspaceApplications(Long workspaceId, String username, boolean isChinese); + + DSSWorkspace getWorkspacesById(Long id); + + List getWorkspaceFavorites(Long workspaceId, String username, boolean isChinese); + + Long addFavorite(String username, Long workspaceId, Long menuApplicationId); + + Long deleteFavorite(String username, Long favouritesId); + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceUserService.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceUserService.java new file mode 100644 index 000000000..a6202690d --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/DSSWorkspaceUserService.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + + + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSUser; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.StaffInfoVO; + +import java.util.List; + + +public interface DSSWorkspaceUserService { + + + void updateWorkspaceUser(List roles, int workspaceId, String userName, String creator); + + void deleteWorkspaceUser(String userName, int workspaceId); + + List listAllDSSUsers(); + + List getAllWorkspaceUsers(int workspaceId); + + List getUserWorkspaceIds(String userName); +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/StaffInfoGetter.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/StaffInfoGetter.java new file mode 100644 index 000000000..38391858f --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/StaffInfoGetter.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service; + +import com.webank.wedatasphere.dss.framework.workspace.bean.StaffInfo; + +import java.util.List; + + +public interface StaffInfoGetter { + + List getAllUsers(); + + String getFullOrgNameByUsername(String username); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSDictionaryServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSDictionaryServiceImpl.java new file mode 100644 index 000000000..88e3980bd --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSDictionaryServiceImpl.java @@ -0,0 +1,120 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSDictionary; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSDictionaryRequestVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DepartmentVO; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSDictionaryMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSDictionaryService; +import com.webank.wedatasphere.dss.framework.workspace.util.DSSDictionaryConstant; +import org.apache.commons.collections.CollectionUtils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import java.util.*; +import java.util.stream.Collectors; + + +@Service +public class DSSDictionaryServiceImpl implements DSSDictionaryService { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSDictionaryServiceImpl.class); + + @Autowired + private DSSDictionaryMapper dssDictionaryMapper; + + @Override + public List getListByParam(DSSDictionaryRequestVO dictionaryRequestVO) { + //查询所在工作空间以及默认空间的开发流程或编码模式配置 + QueryWrapper dictionaryQueryWrapper = new QueryWrapper<>(); + dictionaryQueryWrapper.in("workspace_id", Arrays.asList(dictionaryRequestVO.getWorkspaceId(),0)); + if(StringUtils.isNotBlank(dictionaryRequestVO.getDicKey())){ + dictionaryQueryWrapper.eq("dic_key",dictionaryRequestVO.getDicKey()); + } + if(StringUtils.isNotBlank(dictionaryRequestVO.getParentKey())){ + dictionaryQueryWrapper.eq("parent_key",dictionaryRequestVO.getParentKey()); + } + dictionaryQueryWrapper.orderByAsc("order_num"); + List dictionaries = dssDictionaryMapper.selectList(dictionaryQueryWrapper); + return dictionaries; + } + + @Override + public Map getDicSecondList(DSSDictionaryRequestVO dictionaryRequestVO) { + //一级字典 + List dictionaries = getListByParam(dictionaryRequestVO); + + //二级字典 + List keyList = dictionaries.stream().map(DSSDictionary::getDicKey).collect(Collectors.toList()); + QueryWrapper dictionaryQueryWrapper = new QueryWrapper<>(); + dictionaryQueryWrapper.in("workspace_id", Arrays.asList(dictionaryRequestVO.getWorkspaceId(),0)); + dictionaryQueryWrapper.in("parent_key",keyList); + dictionaryQueryWrapper.orderByAsc("order_num"); + List dictionarieSelectList = dssDictionaryMapper.selectList(dictionaryQueryWrapper); + + //封装返回 + Map> mapList = new HashMap<>(); + for(DSSDictionary dssDictionary : dictionarieSelectList){ + String parentKey = dssDictionary.getParentKey(); + if(mapList.get(parentKey)==null){ + mapList.put(parentKey,new ArrayList<>()); + } + mapList.get(parentKey).add(dssDictionary); + } + Map retMap = new HashMap<>(); + retMap.put("list",dictionaries); + retMap.put("mapList",mapList); + return retMap; + } + + + /** + * 获取空间默认部门 + * @return + */ + @Override + public List getDefaultDepartmentVOList(){ + QueryWrapper dictionaryQueryWrapper = new QueryWrapper<>(); + dictionaryQueryWrapper.eq("workspace_id", 0); + dictionaryQueryWrapper.eq("dic_key", DSSDictionaryConstant.W_WORKSPACE_DEPARTMENT); + List dictionarieList = dssDictionaryMapper.selectList(dictionaryQueryWrapper); + List retList = new ArrayList<>(); + if(CollectionUtils.isEmpty(dictionarieList)){ + return retList; + } + String dicValue = dictionarieList.get(0).getDicValue(); + if(StringUtils.isBlank(dicValue)){ + return retList; + } + String[] tempStrArr = dicValue.split(";"); + DepartmentVO departmentVO = null; + for(int i=0;i getSidebarVOList(String username, Integer workspaceId,boolean isEnglish) { + //查询所在工作空间以及默认空间的侧边栏 + List retList = new ArrayList<>(); + QueryWrapper sidebarQueryWrapper = new QueryWrapper<>(); + sidebarQueryWrapper.in("workspace_id", Arrays.asList(workspaceId,0)); + sidebarQueryWrapper.orderByAsc("order_num"); + List sidebarList = sidebarMapper.selectList(sidebarQueryWrapper); + if(CollectionUtils.isEmpty(sidebarList)){ + return retList; + } + //查询所在工作空间以及默认空间的侧边栏对应的侧边栏-内容 + List sideBarIds = sidebarList.stream().map(Sidebar::getId).collect(Collectors.toList()); + QueryWrapper sidebarQueryContentWrapper = new QueryWrapper<>(); + sidebarQueryContentWrapper.in("workspace_id", Arrays.asList(workspaceId,0)); + sidebarQueryContentWrapper.in("sidebar_id", sideBarIds); + sidebarQueryContentWrapper.orderByAsc("order_num"); + List sidebarContentList = sidebarContentMapper.selectList(sidebarQueryContentWrapper); + + //封装返回数据 + return getSidebarVOS(retList, sidebarList, sidebarContentList,isEnglish); + } + + //封装返回数据 + private List getSidebarVOS(List retList, List sidebarList, List sidebarContentList,boolean isEnglish) { + Map> contentMap = new HashMap<>(); + for(SidebarContent sidebarContent : sidebarContentList){ + Integer sidebarId = sidebarContent.getSidebarId(); + if(contentMap.get(sidebarId)==null){ + contentMap.put(sidebarId,new ArrayList<>()); + } + contentMap.get(sidebarId).add(sidebarContent); + } + for (Sidebar sidebar : sidebarList){ + SidebarVO sidebarVO = new SidebarVO(); + BeanUtils.copyProperties(sidebar,sidebarVO); + international(isEnglish,sidebar, sidebarVO); + List sidebarContentVOList = new ArrayList<>(); + List sidebarContents = contentMap.get(sidebar.getId()); + if(!CollectionUtils.isEmpty(sidebarContents)){ + for (SidebarContent sidebarContent : sidebarContents){ + SidebarContentVO sidebarContentVO = new SidebarContentVO(); + BeanUtils.copyProperties(sidebarContent,sidebarContentVO); + international(isEnglish,sidebarContent,sidebarContentVO); + sidebarContentVOList.add(sidebarContentVO); + } + } + sidebarVO.setContents(sidebarContentVOList); + retList.add(sidebarVO); + } + return retList; + } + + //国际化 + public void international(boolean isEnglish,Sidebar sidebar, SidebarVO sidebarVO){ + if(sidebar==null||!isEnglish){ + return; + } + if(StringUtils.isNotBlank(sidebar.getNameEn())){ + sidebarVO.setName(sidebar.getNameEn()); + } + if(StringUtils.isNotBlank(sidebar.getTitleEn())){ + sidebarVO.setTitle(sidebar.getTitleEn()); + } + } + + //国际化 + public void international(boolean isEnglish,SidebarContent sidebarContent, SidebarContentVO sidebarContentVO){ + if(sidebarContent==null||!isEnglish){ + return; + } + if(StringUtils.isNotBlank(sidebarContent.getNameEn())){ + sidebarContentVO.setName(sidebarContent.getNameEn()); + } + if(StringUtils.isNotBlank(sidebarContent.getTitleEn())){ + sidebarContentVO.setTitle(sidebarContent.getTitleEn()); + } + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSUserServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSUserServiceImpl.java new file mode 100644 index 000000000..69fe64441 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSUserServiceImpl.java @@ -0,0 +1,98 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSUser; +import com.webank.wedatasphere.dss.framework.workspace.bean.LinkisUser; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSUserMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.LinkisUserMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSUserService; +import org.apache.commons.collections.CollectionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import java.util.Date; +import java.util.List; + + +@Service +public class DSSUserServiceImpl implements DSSUserService { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSUserServiceImpl.class); + @Autowired + private DSSUserMapper dssUserMapper; + @Autowired + private LinkisUserMapper linkisUserMapper; + + /** + * 在用户及角色添加到工作空间 之前， + * 判断是否存在 username是否存在dss_user + * 不存在保存 ： 将username以及默认信息分别保存到dss_user、linkis_user两张表中 + * @param userName + */ + @Override + public void saveWorkspaceUser(String userName) { + if (getUserID(userName)==null) { + LOGGER.info("User first enter dss, insert table dss_user"); + DSSUser dssUser = new DSSUser(); + dssUser.setUsername(userName); + dssUser.setName(userName); + dssUser.setFirstLogin(true); + //INSERT INTO dss_user() + // VALUES (#{id},#{username},#{name},#{isFirstLogin}) + dssUserMapper.insert(dssUser); + + /* INSERT INTO linkis_user(`username`,`name`,`create_time`,`update_time`,`is_first_login`) + VALUES (#{username},#{name},now(),now(),1)*/ + LinkisUser linkisUser = new LinkisUser(); + linkisUser.setUsername(userName); + linkisUser.setName(userName); + linkisUser.setCreateTime(new Date()); + linkisUser.setUpdateTime(new Date()); + linkisUser.setFirstLogin(true); + linkisUserMapper.insert(linkisUser); + } + } + + //根据用户名获取userId + @Override + public Long getUserID(String userName) { + DSSUser user = getByUsername(userName); + return user!=null?user.getId():null; + } + + //根据用户名获取userId + @Override + public DSSUser getByUsername(String userName) { + QueryWrapper userQueryWrapper = new QueryWrapper<>(); + userQueryWrapper.eq("username", userName); + List dssUserList = dssUserMapper.selectList(userQueryWrapper); + if (!CollectionUtils.isEmpty(dssUserList)) { + return dssUserList.get(0); + } + return null; + } + + @Override + public boolean isAdminUser(String username) { + DSSUser user = getByUsername(username); + return user!=null?user.getAdmin().booleanValue():false; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceMenuServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceMenuServiceImpl.java new file mode 100644 index 000000000..413712e84 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceMenuServiceImpl.java @@ -0,0 +1,63 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSMenu; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceMenuMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceUserMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceMenuService; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + + +@Service +public class DSSWorkspaceMenuServiceImpl implements DSSWorkspaceMenuService { + + + @Autowired + private DSSWorkspaceMenuMapper dssWorkspaceMenuMapper; + + @Autowired + private DSSWorkspaceUserMapper dssWorkspaceUserMapper; + + @Override + public List getRealComponents(List subMenus, String workspaceId, String username) { + Set dssMenuSet = new HashSet<>(subMenus); + subMenus.forEach(subMenu -> { + if (subMenu.isComponent()){ + int applicationId = subMenu.getApplicationId(); + List roles = dssWorkspaceUserMapper.getRoleInWorkspace(Integer.parseInt(workspaceId),username); + int priv = 0; + for (Integer role : roles) { + Integer rolePriv = dssWorkspaceMenuMapper.getOneCompoentRolePriv(Integer.parseInt(workspaceId), role, applicationId); + if(rolePriv != null){ + priv += rolePriv; + } + } + if (priv == 0){ + dssMenuSet.remove(subMenu); + } + } + }); + return new ArrayList<>(dssMenuSet); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspacePrivServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspacePrivServiceImpl.java new file mode 100644 index 000000000..59c085368 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspacePrivServiceImpl.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspacePrivMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspacePrivService; +import org.apache.commons.math3.util.Pair; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.List; + + +@Service +public class DSSWorkspacePrivServiceImpl implements DSSWorkspacePrivService { + + + @Autowired + private DSSWorkspacePrivMapper dssWorkspacePrivMapper; + + @Override + @Transactional(rollbackFor = Exception.class) + public void updateRoleMenuPriv(int workspaceId, int menuId, String updater, List> pairs) { + pairs.forEach(pair -> { + int roleId = pair.getKey(); + int priv = pair.getValue() ? 1 : 0; + dssWorkspacePrivMapper.updateRoleMenuPriv(workspaceId, menuId, roleId, priv); + }); + } + + @Override + @Transactional(rollbackFor = Exception.class) + public void updateRoleComponentPriv(int workspaceId, int componentId, String username, List> pairs) { + pairs.forEach(pair -> { + int roleId = pair.getKey(); + int priv = pair.getValue() ? 1 : 0; + int count = dssWorkspacePrivMapper.queryCntOfRCP(workspaceId, componentId, roleId); + if (count >= 1) { + dssWorkspacePrivMapper.updateRoleComponentPriv(workspaceId, componentId, roleId, priv); + }else { + dssWorkspacePrivMapper.insertRolComponentPriv(workspaceId, componentId, roleId, priv); + } + }); + } + + @Override + public Integer getRoleId(int workspaceId, String key) { + return dssWorkspacePrivMapper.getRoleId(workspaceId, key); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceRoleServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceRoleServiceImpl.java new file mode 100644 index 000000000..75b2177ee --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceRoleServiceImpl.java @@ -0,0 +1,108 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSRole; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceRoleMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceRoleService; +import com.webank.wedatasphere.dss.framework.workspace.util.DSSWorkspaceConstant; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceDBHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.Date; +import java.util.List; + + +@Service +public class DSSWorkspaceRoleServiceImpl implements DSSWorkspaceRoleService { + + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSWorkspaceRoleServiceImpl.class); + + @Autowired + private DSSWorkspaceRoleMapper dssWorkspaceRoleMapper; + + @Autowired + private WorkspaceDBHelper workspaceDBHelper; + + @Override + @Transactional(rollbackFor = Exception.class) + public void addWorkspaceRole(String roleName, int workspaceId, List menuIds, List componentIds, String username) { + DSSRole dssRole = new DSSRole(); + dssRole.setWorkspaceId(workspaceId); + dssRole.setFrontName(roleName); + dssRole.setName(roleName); + dssRole.setCreateTime(new Date()); + dssRole.setDescription("workspace{ " + workspaceId + " }添加的role"); + dssWorkspaceRoleMapper.addNewRole(dssRole); + List allMenuIds = workspaceDBHelper.getAllMenuIds(); + if (menuIds.size() > 0){ + dssWorkspaceRoleMapper.updateRoleMenu(dssRole.getId(), workspaceId, menuIds, username, 1); + } + if (menuIds.size() < allMenuIds.size()){ + allMenuIds.removeAll(menuIds); + dssWorkspaceRoleMapper.updateRoleMenu(dssRole.getId(), workspaceId, allMenuIds, username, 0); + } + List allComponentIds = workspaceDBHelper.getAllComponentIds(); + if (componentIds.size() > 0){ + dssWorkspaceRoleMapper.updateRoleComponent(dssRole.getId(), workspaceId, componentIds, username, 1); + } + if (componentIds.size() < allComponentIds.size()){ + allComponentIds.removeAll(componentIds); + dssWorkspaceRoleMapper.updateRoleComponent(dssRole.getId(), workspaceId, allComponentIds, username, 0); + } + workspaceDBHelper.retrieveFromDB(); + } + + @Override + public List getRoleInWorkspace(String username, int workspaceId) { + LOGGER.info("get roles for user {} in workspace {}", username, workspaceId); + return dssWorkspaceRoleMapper.getAllRoles(username, workspaceId); + } + + @Override + public Integer getWorkspaceIdByUser(String username) { + List workspaceIds = dssWorkspaceRoleMapper.getWorkspaceIds(username); + Integer defaultWorkspaceId = dssWorkspaceRoleMapper.getDefaultWorkspaceId(DSSWorkspaceConstant.DEFAULT_WORKSPACE_NAME.getValue()); + if (workspaceIds.isEmpty()){ + return defaultWorkspaceId; + } else if (workspaceIds.size() == 1){ + return workspaceIds.get(0); + } else { + workspaceIds.remove(defaultWorkspaceId); + return workspaceIds.get(0); + } + } + + @Override + public int getApiPriv(String username, Integer workspaceId, String roleName, String appName) { + int roleId = dssWorkspaceRoleMapper.getRoleId(roleName, -1); + int componentId = dssWorkspaceRoleMapper.getComponentId(appName); + int count = dssWorkspaceRoleMapper.getCount(workspaceId, componentId, roleId); + if (count >= 1){ + Integer tmpPriv = dssWorkspaceRoleMapper.getPriv(workspaceId, roleId, componentId); + return tmpPriv!=null ? tmpPriv.intValue() : 0; + }else{ + Integer tmpPriv = dssWorkspaceRoleMapper.getPriv(-1,roleId, componentId); + return tmpPriv!=null ? tmpPriv.intValue() : 0; + } + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceServiceImpl.java new file mode 100644 index 000000000..613da6c7c --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceServiceImpl.java @@ -0,0 +1,604 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import static com.webank.wedatasphere.dss.framework.workspace.util.DSSWorkspaceConstant.DEFAULT_DEMO_WORKSPACE_NAME; + +import com.github.pagehelper.PageHelper; +import com.github.pagehelper.PageInfo; +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSFavorite; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSMenu; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspace; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceComponentRolePriv; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceHomepageSetting; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceMenuRolePriv; +import com.webank.wedatasphere.dss.framework.workspace.bean.DSSWorkspaceUser; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.HomepageDemoInstanceVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.HomepageDemoMenuVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.HomepageVideoVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.OnestopMenuAppInstanceVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.OnestopMenuVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.WorkspaceDepartmentVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.dto.response.WorkspaceFavoriteVo; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceComponentPrivVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceHomePageVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceHomepageSettingVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceMenuPrivVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceOverviewVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspacePrivVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceRoleVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceUserVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DepartmentVO; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.SecondaryWorkspaceMenuVO; +import com.webank.wedatasphere.dss.framework.workspace.constant.ApplicationConf; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSComponentRoleMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSMenuRoleMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceHomepageMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceInfoMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceMenuMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceUserMapper; +import com.webank.wedatasphere.dss.framework.workspace.dao.WorkspaceMapper; +import com.webank.wedatasphere.dss.framework.workspace.exception.DSSWorkspaceDuplicateNameException; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSUserService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceMenuService; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceService; +import com.webank.wedatasphere.dss.framework.workspace.service.StaffInfoGetter; +import com.webank.wedatasphere.dss.framework.workspace.util.CommonRoleEnum; +import com.webank.wedatasphere.dss.framework.workspace.util.DSSWorkspaceConstant; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceDBHelper; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceServerConstant; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Date; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +@Service +public class DSSWorkspaceServiceImpl implements DSSWorkspaceService { + private static final Logger LOGGER = LoggerFactory.getLogger(DSSWorkspaceServiceImpl.class); + + @Autowired + private DSSWorkspaceMapper dssWorkspaceMapper; + @Autowired + private DSSWorkspaceUserMapper dssWorkspaceUserMapper; + @Autowired + private DSSWorkspaceMenuMapper dssWorkspaceMenuMapper; + @Autowired + private DSSWorkspaceInfoMapper dssWorkspaceInfoMapper; + @Autowired + private WorkspaceDBHelper workspaceDBHelper; + @Autowired + private StaffInfoGetter staffInfoGetter; + @Autowired + private DSSUserService dssUserService; + @Autowired + private DSSMenuRoleMapper dssMenuRoleMapper; + @Autowired + private DSSWorkspaceHomepageMapper dssWorkspaceHomepageMapper; + @Autowired + private DSSComponentRoleMapper dssComponentRoleMapper; + @Autowired + private DSSWorkspaceMenuService dssWorkspaceMenuService; + @Autowired + private WorkspaceMapper workspaceMapper; + + //创建工作空间 + @Override + @Transactional(rollbackFor = Exception.class) + public int createWorkspace(String workspaceName, String tags, String userName, + String description, String department, String productName) throws ErrorException { + DSSWorkspace dssWorkspace = new DSSWorkspace(); + dssWorkspace.setDescription(description); + dssWorkspace.setName(workspaceName); + dssWorkspace.setLabel(tags); + dssWorkspace.setCreateBy(userName); + dssWorkspace.setDepartment(department); + dssWorkspace.setProduct(productName); + dssWorkspace.setCreateTime(new Date()); + dssWorkspace.setLastUpdateTime(new Date()); + dssWorkspace.setLastUpdateUser(userName); + try{ + dssWorkspaceMapper.createWorkSpace(dssWorkspace); + }catch(Exception e){ + DSSWorkspaceDuplicateNameException exception1 = new DSSWorkspaceDuplicateNameException(50010, "工作空间名重复"); + exception1.initCause(e); + throw exception1; + } + dssWorkspaceUserMapper.insertUser(userName, dssWorkspace.getId(), userName); + dssWorkspaceUserMapper.setUserRoleInWorkspace(dssWorkspace.getId(), CommonRoleEnum.ADMIN.getId(), userName, userName); + dssMenuRoleMapper.insertBatch(workspaceDBHelper.generateDefaultWorkspaceMenuRole(dssWorkspace.getId(), userName)); + dssWorkspaceHomepageMapper.insertBatch(workspaceDBHelper.generateDefaultWorkspaceHomepage(dssWorkspace.getId(), userName)); + dssComponentRoleMapper.insertBatch(workspaceDBHelper.generateDefaultWorkspaceComponentPrivs(dssWorkspace.getId(), userName)); + return dssWorkspace.getId(); + } + + + //把用户及角色添加到工作空间 + @Override + @Transactional(rollbackFor = Exception.class) + public void addWorkspaceUser(List roleIds, int workspaceId, String userName, String creator) { + //根据用户名 从用户表拿到用户id + Long userId = dssUserService.getUserID(userName); + if(userId == null){ + //保存 - dss_user、linkis_user + dssUserService.saveWorkspaceUser(userName); + } + //保存 - dss_workspace_user + dssWorkspaceUserMapper.insertUser(userName, workspaceId, creator); + //保存 - 保存用户角色关系 dss_workspace_user_role + for (Integer roleId : roleIds) { + dssWorkspaceUserMapper.setUserRoleInWorkspace(workspaceId, roleId, userName, creator); + } + } + + //获取所有的工作空间 + @Override + public List getWorkspaces(String userName){ + List workspaces = dssWorkspaceMapper.getWorkspaces(userName); + //用于展示demo的工作空间是不应该返回的,除非用户是管理员 + if(dssWorkspaceUserMapper.isAdmin(userName)) { + return workspaces; + }else{ + //踢掉那个演示demo工作空间 + List retWorkspaces = new ArrayList<>(); + String defaultDemoWorkspaceName = DEFAULT_DEMO_WORKSPACE_NAME.getValue(); + for (DSSWorkspace workspace : workspaces) { + if (!workspace.getName().equals(defaultDemoWorkspaceName)){ + retWorkspaces.add(workspace); + } + } + return retWorkspaces; + } + } + + @Override + public DSSWorkspaceHomePageVO getWorkspaceHomePage(String userName,String moduleName) { + //根据用户名 拿到用户ID + //根据用户id 和工作空间id 拿到 角色id + //根据role id 和工作空间id 拿到 重定向的 url + List tempWorkspaceIds = dssWorkspaceUserMapper.getWorkspaceIds(userName); + List workspaceIds = new ArrayList<>(); + tempWorkspaceIds.stream(). + map(dssWorkspaceInfoMapper::getWorkspaceNameById). + filter(name -> !DEFAULT_DEMO_WORKSPACE_NAME.getValue().equals(name)). + map(dssWorkspaceInfoMapper::getWorkspaceIdByName). + forEach(workspaceIds::add); + DSSWorkspaceHomePageVO dssWorkspaceHomePageVO = new DSSWorkspaceHomePageVO(); + if (workspaceIds.size() == 0){ +// while(userId == null || userId.intValue() < 0){ +// userId = dssWorkspaceUserMapper.getUserID(userName); +// } + int workspaceId = dssWorkspaceInfoMapper.getWorkspaceIdByName(DSSWorkspaceConstant.DEFAULT_WORKSPACE_NAME.getValue()); + dssWorkspaceUserMapper.insertUser(userName, workspaceId, "system"); + dssWorkspaceUserMapper.setUserRoleInWorkspace(workspaceId, CommonRoleEnum.ANALYSER.getId(), userName, "system"); + String homepageUrl = dssWorkspaceUserMapper.getHomepageUrl(workspaceId, CommonRoleEnum.ANALYSER.getId()); + if(ApplicationConf.HOMEPAGE_MODULE_NAME.getValue().equalsIgnoreCase(moduleName)){ + homepageUrl= ApplicationConf.HOMEPAGE_URL.getValue() + workspaceIds.get(0); + } + if(StringUtils.isEmpty(homepageUrl)) { + homepageUrl = "/home" + "?workspaceId=" + workspaceId; + } + dssWorkspaceHomePageVO.setHomePageUrl(homepageUrl); + dssWorkspaceHomePageVO.setWorkspaceId(workspaceId); + dssWorkspaceHomePageVO.setRoleName(CommonRoleEnum.ANALYSER.getName()); + }else if(workspaceIds.size() == 1){ + //只有一个工作空间，那么就返回该工作空间的首页 + List roleIds = dssWorkspaceUserMapper.getRoleInWorkspace(workspaceIds.get(0), userName); + int minRoleId = Collections.min(roleIds); + String homepageUrl = dssWorkspaceUserMapper.getHomepageUrl(workspaceIds.get(0), minRoleId); + if(StringUtils.isNotEmpty(homepageUrl)) { + homepageUrl = "/home" + "?workspaceId=" + workspaceIds.get(0); + }else{ + homepageUrl = "/home" + "?workspaceId=" + workspaceIds.get(0); + } + if(ApplicationConf.HOMEPAGE_MODULE_NAME.getValue().equalsIgnoreCase(moduleName)){ + homepageUrl= ApplicationConf.HOMEPAGE_URL.getValue() + workspaceIds.get(0); + } + dssWorkspaceHomePageVO.setHomePageUrl(homepageUrl); + dssWorkspaceHomePageVO.setWorkspaceId(workspaceIds.get(0)); + dssWorkspaceHomePageVO.setRoleName(workspaceDBHelper.getRoleNameById(minRoleId)); + }else{ + //排除掉默认的默认工作空间bdapWorkspace + String homepageUrl="/workspaceHome?workspaceId=" + workspaceIds.get(0); + if(ApplicationConf.HOMEPAGE_MODULE_NAME.getValue().equalsIgnoreCase(moduleName)){ + homepageUrl= ApplicationConf.HOMEPAGE_URL.getValue() + workspaceIds.get(0); + } + dssWorkspaceHomePageVO.setHomePageUrl(homepageUrl); + } + return dssWorkspaceHomePageVO; + } + + @Override + public List getWorkspaceMenus(String userName, String workspaceId) { + //根据用户及工作空间id获取到用户角色id，根据角色和工作空间id 获取菜单id，根据菜单id 封装1级2级菜单并返回 + List roleIds = dssWorkspaceUserMapper.getRoleInWorkspace(Integer.parseInt(workspaceId), userName); + Set menuIds = new HashSet<>(); + for (Integer roleId : roleIds) { + List menuList = dssWorkspaceMapper.getMenuId(roleId, workspaceId); + menuIds.addAll(menuList); + } + List dssMenuList = new ArrayList<>(); + for (int menuId : menuIds){ + DSSMenu dssMenu = dssWorkspaceMapper.getSpaceMenu(menuId); + dssMenuList.add(dssMenu); + } + return dssMenuList; + } + + @Override + public List getWorkspaceUsers(String workspaceId, String department, String username, + String roleName, int pageNow, int pageSize, List total) { + int roleId = -1; + if (StringUtils.isNotEmpty(roleName)){ + roleId = workspaceDBHelper.getRoleIdByName(roleName); + } + PageHelper.startPage(pageNow, pageSize); + List workspaceUsers = new ArrayList<>(); + try{ + workspaceUsers = dssWorkspaceUserMapper.getWorkspaceUsers(workspaceId, department,username, roleId); + }finally { + PageHelper.clearPage(); + } + PageInfo pageInfo = new PageInfo<>(workspaceUsers); + total.add(pageInfo.getTotal()); + List dssWorkspaceUserVOs = new ArrayList<>(); + for (DSSWorkspaceUser workspaceUser : workspaceUsers) { + List roles = dssWorkspaceUserMapper.getRoleInWorkspace(Integer.parseInt(workspaceId), workspaceUser.getUsername()); + dssWorkspaceUserVOs.add(changeToUserVO(workspaceUser, roles)); + } + return dssWorkspaceUserVOs; + } + + private DSSWorkspaceUserVO changeToUserVO(DSSWorkspaceUser dssWorkspaceUser, List roles){ + DSSWorkspaceUserVO vo = new DSSWorkspaceUserVO(); + String userName = dssWorkspaceUser.getUsername(); + vo.setName(userName); + String orgFullName = staffInfoGetter.getFullOrgNameByUsername(userName); + if (StringUtils.isNotEmpty(orgFullName)){ + try{ + String departmentName = orgFullName.split(WorkspaceServerConstant.DEFAULT_STAFF_SPLIT)[0]; + String officeName = orgFullName.split(WorkspaceServerConstant.DEFAULT_STAFF_SPLIT)[1]; + vo.setDepartment(departmentName); + vo.setOffice(officeName); + }catch(Exception e){ + //LOGGER.warn("fail to get department and office {} ", e.getMessage()); + vo.setDepartment(WorkspaceServerConstant.DEFAULT_DEPARTMENT); + vo.setOffice(WorkspaceServerConstant.DEFAULT_OFFICE); + } + } + vo.setRoles(roles); + vo.setCreator(dssWorkspaceUser.getCreator()); + vo.setJoinTime(dssWorkspaceUser.getJoinTime()); + return vo; + } + + @Override + public List getWorkspaceRoles(int workspaceId) { + return workspaceDBHelper.getRoleVOs(workspaceId); + } + + @Override + public DSSWorkspacePrivVO getWorkspaceMenuPrivs(String workspaceId) { + DSSWorkspacePrivVO dssWorkspacePrivVO = new DSSWorkspacePrivVO(); + dssWorkspacePrivVO.setWorkspaceId(Integer.parseInt(workspaceId)); + dssWorkspacePrivVO.setRoleVOS(workspaceDBHelper.getRoleVOs(Integer.parseInt(workspaceId))); + List dssWorkspaceMenuPrivVOList = new ArrayList<>(); + List dssWorkspaceMenuRolePrivList = dssWorkspaceMapper.getDSSWorkspaceMenuPriv(workspaceId); + Map> map = new HashMap<>(); + for (DSSWorkspaceMenuRolePriv dssWorkspaceMenuPriv : dssWorkspaceMenuRolePrivList){ + int menuId = dssWorkspaceMenuPriv.getMenuId(); + if(!map.containsKey(menuId)){ + map.put(menuId, new ArrayList<>()); + } + map.get(menuId).add(dssWorkspaceMenuPriv); + } + map.forEach((k,v) ->{ + DSSWorkspaceMenuPrivVO vo = new DSSWorkspaceMenuPrivVO(); + vo.setId(k); + vo.setName(workspaceDBHelper.getMenuNameById(k).getFrontName()); + Map menuPrivs = new HashMap<>(); + v.forEach(priv -> { + menuPrivs.put(workspaceDBHelper.getRoleNameById(priv.getRoleId()), priv.getPriv() == 1); + }); + vo.setMenuPrivs(menuPrivs); + dssWorkspaceMenuPrivVOList.add(vo); + }); + dssWorkspacePrivVO.setMenuPrivVOS(dssWorkspaceMenuPrivVOList); + + List dssWorkspaceComponentPrivVOList = new ArrayList<>(); + List dssWorkspaceComponentRolePrivList = + dssWorkspaceMenuMapper.getComponentRolePriv(Integer.parseInt(workspaceId)); + List defaultDssWorkspaceComponentRolePrivList = dssWorkspaceMenuMapper.getDefaultComponentRolePriv(); + for (DSSWorkspaceComponentRolePriv p : defaultDssWorkspaceComponentRolePrivList){ + if (!dssWorkspaceComponentRolePrivList.contains(p)) { + dssWorkspaceComponentRolePrivList.add(p); + } + } + Map> map1 = new HashMap<>(); + for (DSSWorkspaceComponentRolePriv dssWorkspaceComponentRolePriv : dssWorkspaceComponentRolePrivList){ + int componentId = dssWorkspaceComponentRolePriv.getComponentId(); + if(!map1.containsKey(componentId)){ + List tempList = new ArrayList<>(); + tempList.add(dssWorkspaceComponentRolePriv); + map1.put(componentId, tempList); + }else{ + map1.get(componentId).add(dssWorkspaceComponentRolePriv); + } + } + map1.forEach((k,v) ->{ + DSSWorkspaceComponentPrivVO vo = new DSSWorkspaceComponentPrivVO(); + vo.setId(k); + if (workspaceDBHelper.getComponent(k) != null){ + vo.setName(workspaceDBHelper.getComponent(k).getName()); + } else { + vo.setName("linkis"); + } + Map componentPrivs = new HashMap<>(); + v.forEach(priv -> { + componentPrivs.put(workspaceDBHelper.getRoleNameById(priv.getRoleId()), priv.getPriv() == 1); + }); + vo.setComponentPrivs(componentPrivs); + dssWorkspaceComponentPrivVOList.add(vo); + }); + dssWorkspacePrivVO.setComponentPrivVOS(dssWorkspaceComponentPrivVOList); + return dssWorkspacePrivVO; + } + + @Override + public DSSWorkspaceOverviewVO getOverview(String username, int workspaceId, boolean isEnglish) { + DSSWorkspaceOverviewVO dssWorkspaceOverviewVO = new DSSWorkspaceOverviewVO(); + DSSWorkspace dssWorkspace = dssWorkspaceInfoMapper.getWorkspace(workspaceId); + dssWorkspaceOverviewVO.setTitle(dssWorkspace.getName()); + dssWorkspaceOverviewVO.setDescription(dssWorkspace.getDescription()); + return dssWorkspaceOverviewVO; + } + + private SecondaryWorkspaceMenuVO menuChangeToVO(DSSMenu dssMenu){ + SecondaryWorkspaceMenuVO secondaryWorkspaceMenuVO = new SecondaryWorkspaceMenuVO(); + secondaryWorkspaceMenuVO.setId(dssMenu.getId()); + secondaryWorkspaceMenuVO.setName(dssMenu.getFrontName()); + secondaryWorkspaceMenuVO.setUrl(dssMenu.getUrl()); + return secondaryWorkspaceMenuVO; + } + + + @Override + public DSSWorkspaceHomepageSettingVO getWorkspaceHomepageSettings(int workspaceId) { + DSSWorkspaceHomepageSettingVO dssWorkspaceHomepageSettingVO = new DSSWorkspaceHomepageSettingVO(); + + List dssWorkspaceHomepageSettings = dssWorkspaceMenuMapper.getWorkspaceHompageSettings(workspaceId); + List roleHomepageList = new ArrayList<>(); + dssWorkspaceHomepageSettings.forEach(homepage -> { + DSSWorkspaceHomepageSettingVO.RoleHomepage roleHomepage = new DSSWorkspaceHomepageSettingVO.RoleHomepage(); + roleHomepage.setHomepageName(workspaceDBHelper.getHomepageName(homepage.getHomepageUrl())); + roleHomepage.setHomepageUrl(homepage.getHomepageUrl()); + roleHomepage.setRoleId(homepage.getRoleId()); + roleHomepage.setRoleName(workspaceDBHelper.getRoleNameById(homepage.getRoleId())); + roleHomepage.setRoleFrontName(workspaceDBHelper.getRoleFrontName(homepage.getRoleId())); + roleHomepageList.add(roleHomepage); + }); + dssWorkspaceHomepageSettingVO.setRoleHomepages(roleHomepageList); + return dssWorkspaceHomepageSettingVO; + } + + @Override + public String getWorkspaceName(String workspaceId) { + return dssWorkspaceInfoMapper.getWorkspaceNameById(Integer.parseInt(workspaceId)); + } + + @Override + public boolean checkAdmin(String userName) { + return dssWorkspaceUserMapper.isAdmin(userName); + } + + @Override + public List getDepartments() { + List allDepartments = Arrays.asList("WeDataSphere","DataSP", "linkis"); + List departmentVOs = new ArrayList<>(); + int count = 1; + for (String department : allDepartments){ + DepartmentVO departmentVO = new DepartmentVO(); + departmentVO.setFrontName(department); + departmentVO.setName(department); + departmentVO.setId(count); + departmentVOs.add(departmentVO); + count ++; + } + return departmentVOs; + } + + @Override + public List getWorkspaceUsersByRole(int workspaceId, String roleName, List totals, int pageNow, int pageSize) { + int roleId = workspaceDBHelper.getRoleIdByName(roleName); + PageHelper.startPage(pageNow, pageSize); + List workspaceUsers = new ArrayList<>(); + try{ + workspaceUsers = dssWorkspaceUserMapper.getWorkspaceUsersByRole(workspaceId, roleId); + }finally { + PageHelper.clearPage(); + } + PageInfo pageInfo = new PageInfo<>(workspaceUsers); + totals.add(pageInfo.getTotal()); + List dssWorkspaceUserVOs = new ArrayList<>(); + for (DSSWorkspaceUser workspaceUser : workspaceUsers) { + List roles = dssWorkspaceUserMapper.getRoleInWorkspace(workspaceId, workspaceUser.getUsername()); + dssWorkspaceUserVOs.add(changeToUserVO(workspaceUser, roles)); + } + return dssWorkspaceUserVOs; + } + + + @Override + public List getWorkspaces() { + + return workspaceMapper.getWorkspaces(); + } + + @Override + public DSSWorkspace getWorkspacesById(Long id) { + return workspaceMapper.getWorkspaceById(id); + } + + + @Override + public Long addWorkspace(String userName, String name, String department, String label, String description) { + DSSWorkspace dssWorkspace = new DSSWorkspace(); + dssWorkspace.setName(name); + dssWorkspace.setDepartment(department); + dssWorkspace.setDescription(description); + dssWorkspace.setLabel(label); + dssWorkspace.setCreateBy(userName); + dssWorkspace.setSource("create by user"); + dssWorkspace.setLastUpdateUser(userName); + workspaceMapper.addWorkSpace(dssWorkspace); + return (long) dssWorkspace.getId(); + } + + @Override + public boolean existWorkspaceName(String name) { + return !workspaceMapper.findByWorkspaceName(name).isEmpty(); + } + + @Override + public List getWorkSpaceDepartments() { + // TODO: service层和dao层完善 + WorkspaceDepartmentVo dp = new WorkspaceDepartmentVo(); + dp.setId(1L); + dp.setName("应用开发组"); + WorkspaceDepartmentVo di = new WorkspaceDepartmentVo(); + di.setId(2L); + di.setName("平台研发组"); + List departments = new ArrayList<>(); + departments.add(dp); + departments.add(di); + return departments; + } + + @Override + public List getHomepageDemos(boolean isChinese) { + List demoMenuVos = isChinese ? workspaceMapper.getHomepageDemoMenusCn() : workspaceMapper.getHomepageDemoMenusEn(); + for (HomepageDemoMenuVo demoMenuVo : demoMenuVos) { + Long menuId = demoMenuVo.getId(); + List demoInstanceVos = isChinese ? workspaceMapper.getHomepageInstancesByMenuIdCn(menuId) : workspaceMapper.getHomepageInstancesByMenuIdEn(menuId); + demoMenuVo.setDemoInstances(demoInstanceVos); + } + return demoMenuVos; + } + + @Override + public List getHomepageVideos(boolean isChinese) { + return isChinese ? workspaceMapper.getHomepageVideosCn() : workspaceMapper.getHomepageVideosEn(); + } + + @Override + public List getWorkspaceManagements(Long workspaceId, String username, boolean isChinese) { + if (!isAdminUser(workspaceId, username)) { + return new ArrayList<>(); + } + List managementMenuVos = isChinese ? workspaceMapper.getManagementMenuCn() : workspaceMapper.getManagementMenuEn(); + return getMenuAppInstances(managementMenuVos, isChinese); + } + + private List getMenuAppInstances(List menuVos, boolean isChinese) { + for (OnestopMenuVo menuVo : menuVos) { + List appConns = AppConnManager.getAppConnManager().listAppConns(); + + Long menuId = menuVo.getId(); + List menuAppInstanceVos = isChinese ? workspaceMapper.getMenuAppInstancesCn(menuId) : workspaceMapper.getMenuAppInstancesEn(menuId); + for (OnestopMenuAppInstanceVo menuAppInstanceVo : menuAppInstanceVos) { + Map nameAndUrl = new HashMap<>(); + appConns.forEach(appConn -> { + if(appConn.getAppDesc().getAppName().equalsIgnoreCase(menuAppInstanceVo.getName())) { + if (appConn.getAppDesc().getAppInstances().size() == 2) { + appConn.getAppDesc().getAppInstances().forEach(appInstance -> { + + nameAndUrl.put("进入开发中心", appInstance.getBaseUrl()); + + }); + } + } + }); + if(nameAndUrl.size()==0) { + nameAndUrl.put(menuAppInstanceVo.getAccessButton(), menuAppInstanceVo.getAccessButtonUrl()); + } + + menuAppInstanceVo.setNameAndUrls(nameAndUrl); + } + menuVo.setAppInstances(menuAppInstanceVos); + } + return menuVos; + } + + @Override + public List getWorkspaceApplications(Long workspaceId, String username, boolean isChinese) { + List applicationMenuVos = isChinese ? workspaceMapper.getApplicationMenuCn() : workspaceMapper.getApplicationMenuEn(); + return getMenuAppInstances(applicationMenuVos, isChinese); + } + + @Override + public List getWorkspaceFavorites(Long workspaceId, String username, boolean isChinese) { + return isChinese ? workspaceMapper.getWorkspaceFavoritesCn(username, workspaceId) : workspaceMapper.getWorkspaceFavoritesEn(username, workspaceId); + } + + @Override + public Long addFavorite(String username, Long workspaceId, Long menuApplicationId) { + DSSFavorite dssFavorite = new DSSFavorite(); + dssFavorite.setUsername(username); + dssFavorite.setWorkspaceId(workspaceId); + dssFavorite.setMenuApplicationId(menuApplicationId); + // todo: order will from the front end + dssFavorite.setOrder(1); + dssFavorite.setCreateBy(username); + dssFavorite.setLastUpdateUser(username); + workspaceMapper.addFavorite(dssFavorite); + return dssFavorite.getId(); + } + + @Override + public Long deleteFavorite(String username, Long favouritesId) { + workspaceMapper.deleteFavorite(favouritesId); + return favouritesId; + } + + private boolean isAdminUser(Long workspaceId, String username) { + DSSWorkspace workspace = workspaceMapper.getWorkspaceById(workspaceId); + List roles = dssWorkspaceUserMapper.getRoleInWorkspace(workspaceId.intValue(), username); + if(roles != null && roles.size() > 0) { + for (Integer role : roles){ + if (role == 1){ + return true; + } + } + + } + return username != null && workspace != null && username.equals(workspace.getCreateBy()); + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceUserServiceImpl.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceUserServiceImpl.java new file mode 100644 index 000000000..7cdf4864e --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DSSWorkspaceUserServiceImpl.java @@ -0,0 +1,99 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.webank.wedatasphere.dss.framework.workspace.bean.StaffInfo; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.StaffInfoVO; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceUserMapper; +import com.webank.wedatasphere.dss.framework.workspace.service.DSSWorkspaceUserService; +import com.webank.wedatasphere.dss.framework.workspace.service.StaffInfoGetter; +import com.webank.wedatasphere.dss.framework.workspace.util.WorkspaceServerConstant; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.List; +import java.util.stream.Collectors; + + +@Service +public class DSSWorkspaceUserServiceImpl implements DSSWorkspaceUserService { + + private static final Logger LOGGER = LoggerFactory.getLogger(DSSWorkspaceUserServiceImpl.class); + @Autowired + private DSSWorkspaceUserMapper dssWorkspaceUserMapper; + + @Autowired + StaffInfoGetter staffInfoGetter; + + @Override + @Transactional(rollbackFor = Throwable.class) + public void updateWorkspaceUser(List roles, int workspaceId, String userName, String creator) { + dssWorkspaceUserMapper.removeAllRolesForUser(userName, workspaceId); + roles.forEach(role ->{ + dssWorkspaceUserMapper.setUserRoleInWorkspace(workspaceId, role, userName, creator); + }); + } + + @Override + @Transactional(rollbackFor = Exception.class) + public void deleteWorkspaceUser(String userName, int workspaceId) { + dssWorkspaceUserMapper.removeAllRolesForUser(userName, workspaceId); + dssWorkspaceUserMapper.removeUserInWorkspace(userName, workspaceId); + } + + @Override + public List listAllDSSUsers() { + //需要将esb带来的所有用户进行返回 + List staffInfos = staffInfoGetter.getAllUsers(); + return staffInfos.stream().map(this::staffToDSSUser).collect(Collectors.toList()); + } + + private StaffInfoVO staffToDSSUser(StaffInfo staffInfo){ + StaffInfoVO staffInfoVO = new StaffInfoVO(); + String orgFullName = staffInfo.getOrgFullName(); + if (StringUtils.isNotEmpty(orgFullName)){ + try{ + String departmentName = orgFullName.split(WorkspaceServerConstant.DEFAULT_STAFF_SPLIT)[0]; + String officeName = orgFullName.split(WorkspaceServerConstant.DEFAULT_STAFF_SPLIT)[1]; + staffInfoVO.setDepartment(departmentName); + staffInfoVO.setOffice(officeName); + }catch(Exception e){ + //LOGGER.warn("fail to get department and office {} ", e.getMessage()); + staffInfoVO.setDepartment(WorkspaceServerConstant.DEFAULT_DEPARTMENT); + staffInfoVO.setOffice(WorkspaceServerConstant.DEFAULT_OFFICE); + } + } + staffInfoVO.setUsername(staffInfo.getEnglishName()); + staffInfoVO.setId(staffInfo.getId()); + return staffInfoVO; + } + + @Override + public List getAllWorkspaceUsers(int workspaceId) { + return dssWorkspaceUserMapper.getAllWorkspaceUsers(workspaceId); + } + + @Override + public List getUserWorkspaceIds(String userName){ + List tempWorkspaceIds = dssWorkspaceUserMapper.getWorkspaceIds(userName); + return tempWorkspaceIds; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DefaultStaffInfoGetter.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DefaultStaffInfoGetter.java new file mode 100644 index 000000000..24e6d6b02 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/service/impl/DefaultStaffInfoGetter.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.service.impl; + +import com.google.common.collect.Lists; +import com.webank.wedatasphere.dss.framework.workspace.bean.StaffInfo; +import com.webank.wedatasphere.dss.framework.workspace.service.StaffInfoGetter; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.stereotype.Component; + +import java.util.List; + + + +public class DefaultStaffInfoGetter implements StaffInfoGetter { + + @Override + public List getAllUsers() { + return Lists.newArrayList(new StaffInfo("1","hadoop", "WeDataSphere")); + } + + @Override + public String getFullOrgNameByUsername(String username) { + return "WeDataSphere"; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/ApplicationUtils.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/ApplicationUtils.java new file mode 100644 index 000000000..2ab47b549 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/ApplicationUtils.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.net.URLEncoder; + + +public class ApplicationUtils { + + private static final Logger LOGGER = LoggerFactory.getLogger(ApplicationUtils.class); + + private static final String REDIRECT_FORMAT = "%s?redirect=%s&dssurl=${dssurl}&cookies=${cookies}"; + + private static String URLEndoder(String str){ + try { + return URLEncoder.encode(str,"utf-8"); + } catch (UnsupportedEncodingException e) { + LOGGER.warn("endoe failed:",e); + return str; + } + } + + public static String redirectUrlFormat(String redirectUrl, String url){ + return String.format(REDIRECT_FORMAT,redirectUrl,URLEndoder(url)); + } + + public static void main(String[] args) throws DSSErrorException { + System.out.println(redirectUrlFormat("http://127.0..0.1:8090/qualitis/api/v1/redirect","http://127.0..0.1:8090/#/projects/list?id={projectId}&flow=true")); + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/CommonRoleEnum.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/CommonRoleEnum.java new file mode 100644 index 000000000..d2d9e370a --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/CommonRoleEnum.java @@ -0,0 +1,64 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + + +public enum CommonRoleEnum { + /** + * 通用角色的枚举 + */ + ADMIN(1, "admin", "管理员"), + MAINTENANCE(2, "maintaince","运维人员"), + DEVELOPER(3, "developer", "开发人员"), + ANALYSER(4, "analyser", "分析用户"), + OPERATOR(5, "operator", "运营用户"), + BOSS(6, "boss", "领导"); + + private int id; + private String name; + private String frontName; + + private CommonRoleEnum(int id, String name, String frontName){ + this.id = id; + this.name = name; + this.frontName = frontName; + } + + public int getId() { + return id; + } + + public void setId(int id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getFrontName() { + return frontName; + } + + public void setFrontName(String frontName) { + this.frontName = frontName; + } +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/DSSDictionaryConstant.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/DSSDictionaryConstant.java new file mode 100644 index 000000000..ccac16879 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/DSSDictionaryConstant.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + + +public class DSSDictionaryConstant { + public static final String W_WORKSPACE_DEPARTMENT = "w_workspace_department"; +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/DSSWorkspaceConstant.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/DSSWorkspaceConstant.java new file mode 100644 index 000000000..6d8537e5b --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/DSSWorkspaceConstant.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public interface DSSWorkspaceConstant { + public static final String WORKSPACE_ID_STR = "workspaceId"; + + public static final CommonVars DEFAULT_WORKSPACE_NAME + = CommonVars.apply("wds.dss.workspace.default.name", "bdapWorkspace"); + + public static final String COMMON_URL_PREFIX = "/workspaceHome/common/"; + + CommonVars DEFAULT_DEMO_WORKSPACE_NAME = CommonVars.apply("wds.dss.default.demo.workspace", "WDS_DSS_DEMO"); + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/RestfulUtils.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/RestfulUtils.java new file mode 100644 index 000000000..e019a774c --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/RestfulUtils.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + +import com.webank.wedatasphere.linkis.server.Message; +import org.apache.commons.math3.util.Pair; + + +import javax.ws.rs.core.Response; +import java.util.Arrays; + + +public class RestfulUtils { + + public static Response dealError(String reason){ + Message message = Message.error(reason); + return Message.messageToResponse(message); + } + + public static Response dealOk(String msg){ + Message message = Message.ok(msg); + return Message.messageToResponse(message); + } + + + + @SafeVarargs + public static Response dealOk(String msg, Pair... data){ + Message message = Message.ok(msg); + Arrays.stream(data).forEach(p -> message.data(p.getKey(), p.getValue())); + return Message.messageToResponse(message); + } + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/RestulHelper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/RestulHelper.java new file mode 100644 index 000000000..9cd51e7fb --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/RestulHelper.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + + +import com.webank.wedatasphere.dss.framework.workspace.exception.DSSWorkspaceLoginFailException; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; + + +@Component +public class RestulHelper { + + @Autowired + private WorkspaceDBHelper workspaceDBHelper; + + public String getLoginUser(HttpServletRequest request)throws ErrorException { + try{ + return SecurityFilter.getLoginUsername(request); + }catch(Exception e){ + throw new DSSWorkspaceLoginFailException(80013, "You are not logged in"); + } + } + + + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceDBHelper.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceDBHelper.java new file mode 100644 index 000000000..f988d47ee --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceDBHelper.java @@ -0,0 +1,337 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + + +import com.webank.wedatasphere.dss.framework.workspace.bean.*; +import com.webank.wedatasphere.dss.framework.workspace.bean.vo.DSSWorkspaceRoleVO; +import com.webank.wedatasphere.dss.framework.workspace.constant.ApplicationConf; +import com.webank.wedatasphere.dss.framework.workspace.dao.DSSWorkspaceRoleMapper; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; +import java.util.*; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; + + +@Component +public class WorkspaceDBHelper { + + private static final Logger LOGGER = LoggerFactory.getLogger(WorkspaceDBHelper.class); + + @Autowired + private DSSWorkspaceRoleMapper dssWorkspaceRoleMapper; + + + + private List dssRoles; + + private final Object lock = new Object(); + + private List dssMenus; + + private List dssWorkspaceMenuComponentUrls; + + private List dssWorkspaceComponents; + + private List dssApplicationBeans; + + private Map dssHomepageNameMap = new HashMap<>(); + + + @PostConstruct + public void init(){ + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (lock){ + dssRoles = dssWorkspaceRoleMapper.getRoles(); + } + } + },0, 1, TimeUnit.MINUTES); + + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (lock){ + dssMenus = dssWorkspaceRoleMapper.getMenus(); + } + } + },0, 1, TimeUnit.MINUTES); + + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (lock){ + dssWorkspaceMenuComponentUrls = dssWorkspaceRoleMapper.getMenuComponentUrl(); + } + } + },0, 1, TimeUnit.MINUTES); + + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (lock){ + dssWorkspaceComponents = dssWorkspaceRoleMapper.getComponents(); + } + } + },0 ,1, TimeUnit.MINUTES); + + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (lock){ + dssApplicationBeans = dssWorkspaceRoleMapper.getDSSApplications(); + } + } + }, 0, 1, TimeUnit.MINUTES); + + + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (lock){ + String defaultUrl = "/workspace"; + String maintenanceUrl = "/workspaceHome/scheduleCenter"; + String developerUrl = + dssApplicationBeans.stream(). + filter(dssApplicationBean -> "workflow".equals(dssApplicationBean.getName().toLowerCase())). + findFirst().map(DSSApplicationBean::getHomepageUrl).orElse(defaultUrl); + String analyserUrl = dssApplicationBeans.stream(). + filter(dssApplicationBean -> "linkis".equals(dssApplicationBean.getName().toLowerCase())). + findFirst().map(DSSApplicationBean::getHomepageUrl).orElse(defaultUrl); + dssHomepageNameMap.put("/workspace", "工作空间首页"); + dssHomepageNameMap.put(developerUrl, "工作流开发"); + dssHomepageNameMap.put(analyserUrl, "意书(Scriptis)"); + dssHomepageNameMap.put(maintenanceUrl, "调度中心首页"); + } + } + }, 1, 1, TimeUnit.MINUTES); + } + + public void retrieveFromDB(){ + synchronized (lock){ + dssApplicationBeans = dssWorkspaceRoleMapper.getDSSApplications(); + dssWorkspaceComponents = dssWorkspaceRoleMapper.getComponents(); + dssWorkspaceMenuComponentUrls = dssWorkspaceRoleMapper.getMenuComponentUrl(); + dssMenus = dssWorkspaceRoleMapper.getMenus(); + dssRoles = dssWorkspaceRoleMapper.getRoles(); + } + } + + + + public List generateDefaultWorkspaceMenuRole(int workspaceId, String username){ + List list = new ArrayList<>(); + Date date = new Date(System.currentTimeMillis()); + + list.add(new DSSWorkspaceMenuRole(workspaceId, 2,1,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 3,1,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 28,1,1, date, username)); + + list.add(new DSSWorkspaceMenuRole(workspaceId, 2,2,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 3,2,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 28,2,0, date, username)); + + list.add(new DSSWorkspaceMenuRole(workspaceId, 2,3,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 3,3,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 28,3,0, date, username)); + + list.add(new DSSWorkspaceMenuRole(workspaceId, 2,4,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 3,4,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 28,4,0, date, username)); + + list.add(new DSSWorkspaceMenuRole(workspaceId, 2,5,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 3,5,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 28,5,0, date, username)); + + list.add(new DSSWorkspaceMenuRole(workspaceId, 2,6,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 3,6,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 28,6,0, date, username)); + + if(StringUtils.isNotBlank(ApplicationConf.ESB_APPID)){ + list.add(new DSSWorkspaceMenuRole(workspaceId, 1,1,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 1,2,1, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 1,3,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 1,4,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 1,5,0, date, username)); + list.add(new DSSWorkspaceMenuRole(workspaceId, 1,6,1, date, username)); + } + return list; + } + + + public List generateDefaultWorkspaceHomepage(int workspaceId, String username){ + String defaultUrl = "/workspace"; + String maintenanceUrl = "/workspaceHome/scheduleCenter"; + String developerUrl = + dssApplicationBeans.stream(). + filter(dssApplicationBean -> "workflow".equals(dssApplicationBean.getName().toLowerCase())). + findFirst().map(DSSApplicationBean::getHomepageUrl).orElse(defaultUrl); + String analyserUrl = dssApplicationBeans.stream(). + filter(dssApplicationBean -> "linkis".equals(dssApplicationBean.getName().toLowerCase())). + findFirst().map(DSSApplicationBean::getHomepageUrl).orElse(defaultUrl); + List dssWorkspaceHomepages = new ArrayList<>(); + Date date = new Date(System.currentTimeMillis()); + dssWorkspaceHomepages.add(new DSSWorkspaceHomepage(workspaceId, 1, defaultUrl, date)); + dssWorkspaceHomepages.add(new DSSWorkspaceHomepage(workspaceId, 2, analyserUrl, date)); + dssWorkspaceHomepages.add(new DSSWorkspaceHomepage(workspaceId, 3, analyserUrl, date)); + dssWorkspaceHomepages.add(new DSSWorkspaceHomepage(workspaceId, 4, analyserUrl, date)); + dssWorkspaceHomepages.add(new DSSWorkspaceHomepage(workspaceId, 5, defaultUrl, date)); + dssWorkspaceHomepages.add(new DSSWorkspaceHomepage(workspaceId, 6, defaultUrl, date)); + return dssWorkspaceHomepages; + } + + + public List generateDefaultWorkspaceComponentPrivs(int workspaceId, + String username){ + List dssWorkspaceComponentPrivs = new ArrayList<>(); + Date updateTime = new Date(System.currentTimeMillis()); + //admin + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 1, 1, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 1, 2, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 1, 4, 1, updateTime, username)); + //运维 + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 2, 1, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 2, 2, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 2, 4, 1, updateTime, username)); + //开发人员 + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 3, 1, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 3, 2, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 3, 4, 1, updateTime, username)); + //分析人员 + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 4, 1, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 4, 2, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 4, 4, 1, updateTime, username)); + //运营人员 + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 5, 1, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 5, 2, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 5, 4, 1, updateTime, username)); + //领导 + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 6, 1, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 6, 2, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 6, 4, 1, updateTime, username)); + if(StringUtils.isNotBlank(ApplicationConf.ESB_APPID)){ + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 1, 3, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 2, 3, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 3, 3, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 4, 3, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 5, 3, 1, updateTime, username)); + dssWorkspaceComponentPrivs.add(new DSSWorkspaceComponentPriv(workspaceId, 6, 3, 1, updateTime, username)); + } + return dssWorkspaceComponentPrivs; + } + + public List getRoles(){ + return this.dssRoles; + } + + public List getRoles(int workspaceId){ + return this.dssRoles.stream(). + filter(dssRole -> dssRole.getWorkspaceId() == workspaceId || dssRole.getWorkspaceId() == -1). + collect(Collectors.toList()); + } + + + + public List getComponentUrlsById(int componentId){ + Optional optional = dssWorkspaceMenuComponentUrls. + stream().filter(obj -> obj.getMenuId() == componentId). + findFirst(); + if (optional.isPresent()){ + int applicationId = optional.get().getDssApplicationId(); + String homepageUrl = dssWorkspaceRoleMapper.getEntryUrl(applicationId); + return Collections.singletonList(homepageUrl); + }else{ + return null; + } + } + + + + public String getRoleFrontName(int roleId) { + DSSRole role = dssRoles.stream().filter(dssRole -> dssRole.getId() == roleId).findFirst().orElse(null); + if (role != null){ + return role.getFrontName(); + }else{ + return null; + } + } + + public int getRoleIdByName(String roleName){ + if (StringUtils.isBlank(roleName)){ + return -1; + } + return dssRoles.stream(). + filter(dssRole -> roleName.equals(dssRole.getName())). + findFirst(). + map(DSSRole::getId). + orElse(-1); + } + + + public List getRoleVOs(int workspaceId) { + return getRoles(workspaceId).stream().map(this::changeToRoleVO).collect(Collectors.toList()); + } + + private DSSWorkspaceRoleVO changeToRoleVO(DSSRole dssRole){ + DSSWorkspaceRoleVO vo = new DSSWorkspaceRoleVO(); + vo.setRoleFrontName(dssRole.getFrontName()); + vo.setRoleId(dssRole.getId()); + vo.setRoleName(dssRole.getName()); + return vo; + } + + public DSSMenu getMenuNameById(int menuId){ + if(dssMenus.stream().anyMatch(dssMenu -> dssMenu.getId() == menuId)){ + return dssMenus.stream().filter(dssMenu -> dssMenu.getId() == menuId).findFirst().get(); + }else{ + return null; + } + } + + + public String getRoleNameById(int roleId) { + return dssRoles.stream().filter(dssRole -> dssRole.getId() == roleId).findFirst().orElse(new DSSRole()).getName(); + } + + public DSSApplicationBean getComponent(int componentId) { + return dssApplicationBeans.stream(). + filter(dssApplicationBean -> dssApplicationBean.getId() == componentId). + findFirst(). + orElse(null); + } + + public String getHomepageName(String homepage){ + return this.dssHomepageNameMap.get(homepage); + } + + public List getAllMenuIds() { + return dssMenus.stream().filter(dssMenu -> dssMenu.getLevel() == 1).map(DSSMenu::getId).collect(Collectors.toList()); + } + + public List getAllComponentIds() { + return dssWorkspaceComponents.stream().map(DSSWorkspaceComponent::getId).collect(Collectors.toList()); + } + +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceServerConstant.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceServerConstant.java new file mode 100644 index 000000000..ab75cda43 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceServerConstant.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + + +public interface WorkspaceServerConstant { + String DEFAULT_DEPARTMENT = "基础科技部"; + String DEFAULT_OFFICE = "大数据平台室"; + String DEFAULT_STAFF_SPLIT = "-"; +} diff --git a/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceSpringConfig.java b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceSpringConfig.java new file mode 100644 index 000000000..43ac0fbb1 --- /dev/null +++ b/dss-framework/dss-framework-workspace-server/src/main/java/com/webank/wedatasphere/dss/framework/workspace/util/WorkspaceSpringConfig.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.framework.workspace.util; + +import com.webank.wedatasphere.dss.framework.workspace.service.StaffInfoGetter; +import com.webank.wedatasphere.dss.framework.workspace.service.impl.DefaultStaffInfoGetter; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; +import org.springframework.stereotype.Component; + + +@Configuration +public class WorkspaceSpringConfig { + + @Bean + @ConditionalOnMissingBean + public StaffInfoGetter getStaffInfoGetter(){ + return new DefaultStaffInfoGetter(); + } + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/pom.xml b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/pom.xml new file mode 100644 index 000000000..d39a58479 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/pom.xml @@ -0,0 +1,114 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-framework-orchestrator-publish + + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + compile + + + linkis-common + com.webank.wedatasphere.linkis + + + + + + com.webank.wedatasphere.dss + dss-orchestrator-db + ${dss.version} + compile + + + + com.webank.wedatasphere.dss + dss-orchestrator-loader + ${dss.version} + compile + + + org.springframework + spring-tx + ${spring.version} + provided + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-sender-service + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + org.springframework + spring-context + ${spring.version} + provided + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + org.apache.maven.plugins + maven-jar-plugin + + + + + + \ No newline at end of file diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ConversionDSSOrchestratorPlugin.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ConversionDSSOrchestratorPlugin.java new file mode 100644 index 000000000..7db889a25 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ConversionDSSOrchestratorPlugin.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish; + +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import java.util.List; + + +public interface ConversionDSSOrchestratorPlugin extends DSSOrchestratorPlugin { + + ResponseOperateOrchestrator convert(String userName, + DSSProject project, + Workspace workspace, + List orcAppIdList, + List dssLabels); + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ExportDSSOrchestratorPlugin.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ExportDSSOrchestratorPlugin.java new file mode 100644 index 000000000..74de660e0 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ExportDSSOrchestratorPlugin.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import java.util.List; +import java.util.Map; + + +public interface ExportDSSOrchestratorPlugin extends DSSOrchestratorPlugin { + + + Map exportOrchestrator(String userName, + String workspaceName, + Long orchestratorId, + Long orcVersionId, + String projectName, + List dssLabels, + boolean addOrcVersion, Workspace workspace) throws DSSErrorException; + + Long orchestratorVersionIncrease(Long orcId, + String userName, + String comment, + String workspaceName, + DSSOrchestratorInfo dssOrchestratorInfo, + String projectName, + List dssLabels) throws DSSErrorException; + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ImportDSSOrchestratorPlugin.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ImportDSSOrchestratorPlugin.java new file mode 100644 index 000000000..64ccffc87 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/ImportDSSOrchestratorPlugin.java @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import java.io.IOException; +import java.util.List; + + +public interface ImportDSSOrchestratorPlugin extends DSSOrchestratorPlugin { + + /** + * 导入Orchestrator + * @param userName 用户名 + * @param workspaceName 工作空间名 + * @param projectName 工程名 + * @param projectId 工程id + * @param resourceId bml resourceId + * @param version bml version + * @param dssLabels dss标签 + * @param workspace + * @return + * @throws DSSErrorException + * @throws IOException + * @throws ExternalOperationFailedException + */ + Long importOrchestrator(String userName, + String workspaceName, + String projectName, + Long projectId, + String resourceId, + String version, + List dssLabels, + Workspace workspace) throws Exception; + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/conf/DSSOrchestratorConf.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/conf/DSSOrchestratorConf.java new file mode 100644 index 000000000..99544e26a --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/conf/DSSOrchestratorConf.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.conf; + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public class DSSOrchestratorConf { + public static final CommonVars DSS_EXPORT_ENV = CommonVars.apply("wds.dss.server.export.env", "dev"); +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ConversionDSSOrchestratorPluginImpl.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ConversionDSSOrchestratorPluginImpl.java new file mode 100644 index 000000000..feaec126f --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ConversionDSSOrchestratorPluginImpl.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.impl; + +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestConvertOrchestrations; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.AbstractDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.publish.ConversionDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.util.List; + + +public class ConversionDSSOrchestratorPluginImpl extends AbstractDSSOrchestratorPlugin implements ConversionDSSOrchestratorPlugin { + + @Override + public ResponseOperateOrchestrator convert(String userName, + DSSProject project, + Workspace workspace, + List orcAppIdList, + List dssLabels) { + //1、导出第三方应用信息，如工作流、Visualis、Qualities + RequestConvertOrchestrations requestConvertOrchestrator = new RequestConvertOrchestrations(); + requestConvertOrchestrator.setOrcAppIds(orcAppIdList); + requestConvertOrchestrator.setProject(project); + requestConvertOrchestrator.setWorkspace(workspace); + requestConvertOrchestrator.setDSSLabels(dssLabels); + requestConvertOrchestrator.setUserName(userName); + Sender sender = DSSSenderServiceFactory.getOrCreateServiceInstance().getWorkflowSender(dssLabels); + ResponseOperateOrchestrator response = (ResponseOperateOrchestrator) sender.ask(requestConvertOrchestrator); + return response; + } +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ExportDSSOrchestratorPluginImpl.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ExportDSSOrchestratorPluginImpl.java new file mode 100644 index 000000000..fa016b688 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ExportDSSOrchestratorPluginImpl.java @@ -0,0 +1,222 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.impl; + +import static com.webank.wedatasphere.dss.common.utils.ZipHelper.zipExportProject; + +import com.webank.wedatasphere.dss.common.entity.IOType; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.core.exception.DSSOrchestratorErrorException; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.AbstractDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCopyRequestRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorCopyResponseRef; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorExportRequestRef; +import com.webank.wedatasphere.dss.orchestrator.core.service.BMLService; +import com.webank.wedatasphere.dss.orchestrator.core.utils.OrchestratorUtils; +import com.webank.wedatasphere.dss.orchestrator.db.dao.OrchestratorMapper; +import com.webank.wedatasphere.dss.orchestrator.loader.utils.OrchestratorLoaderUtils; +import com.webank.wedatasphere.dss.orchestrator.publish.ExportDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.publish.io.export.MetaExportService; +import com.webank.wedatasphere.dss.standard.app.development.service.RefCRUDService; +import com.webank.wedatasphere.dss.standard.app.development.service.RefExportService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractResponseRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AppConnRefFactoryUtils; +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Field; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.Date; +import java.util.List; +import java.util.Map; +import javafx.util.Pair; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; +import org.springframework.transaction.annotation.Transactional; + + +@Component +public class ExportDSSOrchestratorPluginImpl extends AbstractDSSOrchestratorPlugin implements ExportDSSOrchestratorPlugin { + + static final String DEFAULT_ORC_NAME = "default_orc"; + + @Autowired + private BMLService bmlService; + @Autowired + private OrchestratorMapper orchestratorMapper; + @Autowired + private MetaExportService metaExportService; + @Autowired + private ContextService contextService; + + @Override + @Transactional(rollbackFor = Exception.class) + public Map exportOrchestrator(String userName, String workspaceName, Long orchestratorId, Long orcVersionId, String projectName, + List dssLabels, boolean addOrcVersion, Workspace workspace) throws DSSErrorException { + //1、导出info信息 + if (orcVersionId == null || orcVersionId < 0){ + LOGGER.info("orchestratorVersionId is {}", orcVersionId); + //最简单的就是通过orcId来找到最新的versionId + orcVersionId = orchestratorMapper.findLatestOrcVersionId(orchestratorId); + } + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getOrchestratorVersion(orcVersionId); + DSSOrchestratorInfo dssOrchestratorInfo = orchestratorMapper.getOrchestrator(dssOrchestratorVersion.getOrchestratorId()); + if (dssOrchestratorInfo != null) { + String orcExportSaveBasePath = IoUtils.generateIOPath(userName, DEFAULT_ORC_NAME, ""); + try { + Files.createDirectories(Paths.get(orcExportSaveBasePath).toAbsolutePath().normalize()); + //标记当前导出为project导出 + IoUtils.generateIOType(IOType.ORCHESTRATOR, orcExportSaveBasePath); + //标记当前导出环境env + IoUtils.generateIOEnv(orcExportSaveBasePath); + metaExportService.export(dssOrchestratorInfo, orcExportSaveBasePath); + } catch (IOException e) { + LOGGER.error("Failed to export metaInfo in orchestrator server for orc({}) in version {}.", orchestratorId, orcVersionId, e); + DSSExceptionUtils.dealErrorException(60099, "Failed to export metaInfo in orchestrator server.", e, DSSOrchestratorErrorException.class); + } + LOGGER.info("{} 开始导出Orchestrator: {} 版本ID为: {} ", userName, dssOrchestratorInfo.getName(), orcVersionId); + + //2、导出第三方应用信息，如工作流、Visualis、Qualities + Pair standardPair = OrchestratorLoaderUtils + .getOrcDevelopStandard(userName, workspaceName, dssOrchestratorInfo, dssLabels); + if (null != standardPair) { + RefExportService refExportService = standardPair.getValue().getRefExportService(standardPair.getKey()); + + if (refExportService == null) { + LOGGER.error("for {} to export orchestrator {} refExportService is null", userName, dssOrchestratorInfo.getName()); + DSSExceptionUtils.dealErrorException(60089, "refExportService is null", DSSErrorException.class); + } + OrchestratorExportRequestRef orchestratorExportRequestRef = + AppConnRefFactoryUtils.newAppConnRef(OrchestratorExportRequestRef.class, + refExportService.getClass().getClassLoader(), dssOrchestratorInfo.getType()); + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("orchestrator Export Request Ref class is {}", orchestratorExportRequestRef.getClass()); + } + orchestratorExportRequestRef.setAppId(dssOrchestratorVersion.getAppId()); + orchestratorExportRequestRef.setProjectId(dssOrchestratorVersion.getProjectId()); + orchestratorExportRequestRef.setProjectName(projectName); + orchestratorExportRequestRef.setUserName(userName); + orchestratorExportRequestRef.setWorkspace(workspace); + orchestratorExportRequestRef.setDSSLabels(dssLabels); + AbstractResponseRef responseRef = (AbstractResponseRef) refExportService.getRefExportOperation(). + exportRef(orchestratorExportRequestRef); + String resourceId = responseRef.getValue("resourceId").toString(); + String version = responseRef.getValue("version").toString(); + bmlService.downloadToLocalPath(userName, resourceId, version, orcExportSaveBasePath + "orc_flow.zip"); + } + + //打包导出工程 + String exportPath = zipExportProject(orcExportSaveBasePath); + + //3、打包新的zip包上传BML + InputStream inputStream = bmlService.readLocalResourceFile(userName, exportPath); + Map resultMap = bmlService.upload(userName, inputStream, + dssOrchestratorInfo.getName() + ".OrcExport", projectName); + + //4、判断导出后是否改变Orc的版本 + if (addOrcVersion) { + Long orcIncreaseVersionId = orchestratorVersionIncrease(dssOrchestratorInfo.getId(), + userName, dssOrchestratorInfo.getComment(), + workspaceName, dssOrchestratorInfo, projectName, dssLabels); + resultMap.put("orcVersionId", orcIncreaseVersionId); + } else { + resultMap.put("orcVersionId", orcVersionId); + } + return resultMap; + //4、返回BML存储信息 + } else { + throw new DSSErrorException(90038, "该Orchestrator的版本号不存在，请检查版本号是否正确"); + } + } + + @Override + public Long orchestratorVersionIncrease(Long orcId, + String userName, + String comment, + String workspaceName, + DSSOrchestratorInfo dssOrchestratorInfo, + String projectName, + List dssLabels) throws DSSErrorException { + //对于导出来说,json需替换 subflowID + DSSOrchestratorVersion dssOrchestratorVersion = orchestratorMapper.getLatestOrchestratorVersionById(orcId); + + // TODO: 2020/3/25 set updator(userID 修改为userName后) + dssOrchestratorVersion.setUpdateTime(new Date()); + Long oldOrcVersionId = dssOrchestratorVersion.getId(); + String oldOrcVersion = dssOrchestratorVersion.getVersion(); + dssOrchestratorVersion.setId(null); + // 如果是project发布 ,version都是01,如果是工作流发布,version + 1 + dssOrchestratorVersion.setVersion(OrchestratorUtils.increaseVersion(oldOrcVersion)); + + //发布的comment应该更新到上个版本,而当前最新版本的comment应该修改为release from.. + dssOrchestratorVersion.setComment(String.format("release from version %s", oldOrcVersion)); + //更新老版本的comment + DSSOrchestratorVersion updateCommentVersion = new DSSOrchestratorVersion(); + updateCommentVersion.setId(oldOrcVersionId); + String realComment = comment != null ? comment : "release comment"; + updateCommentVersion.setComment(realComment); + orchestratorMapper.updateOrchestratorVersion(updateCommentVersion); + + //要求AppConn对应第三方应用拷贝一个新的app出来关联，如工作流，需要新建一个新的工作流进行关联。 + + Pair standardPair = OrchestratorLoaderUtils.getOrcDevelopStandard(userName, workspaceName, dssOrchestratorInfo, dssLabels); + if(standardPair == null){ + LOGGER.error("dev Process Service is null"); + throw new DSSOrchestratorErrorException(61105, "dev Process Service is null"); + } + RefCRUDService refcrudservice =standardPair.getValue().getRefCRUDService (standardPair.getKey()); + if (null != refcrudservice) { + try { + OrchestratorCopyRequestRef orchestratorCopyRequestRef = + AppConnRefFactoryUtils.newAppConnRef(OrchestratorCopyRequestRef.class, + refcrudservice.getClass().getClassLoader(), dssOrchestratorInfo.getAppConnName()); + orchestratorCopyRequestRef.setCopyOrcAppId(dssOrchestratorVersion.getAppId()); + orchestratorCopyRequestRef.setCopyOrcVersionId(dssOrchestratorVersion.getOrchestratorId()); + orchestratorCopyRequestRef.setUserName(userName); + orchestratorCopyRequestRef.setProjectName(projectName); + + //5、生成上下文ContextId + String contextId = contextService.createContextID(workspaceName, projectName, dssOrchestratorInfo.getName(), dssOrchestratorVersion.getVersion(), userName); + dssOrchestratorVersion.setContextId(contextId); + LOGGER.info("Create a new ContextId for increase: {} ", contextId); + orchestratorCopyRequestRef.setContextID(contextId); + OrchestratorCopyResponseRef orchestratorCopyResponseRef = + (OrchestratorCopyResponseRef) refcrudservice.getRefCopyOperation().copyRef(orchestratorCopyRequestRef); + dssOrchestratorVersion.setAppId(orchestratorCopyResponseRef.getCopyTargetAppId()); + dssOrchestratorVersion.setContent(orchestratorCopyResponseRef.getCopyTargetContent()); + //update appconn node contextId + dssOrchestratorVersion.setFormatContextId(contextId); + orchestratorMapper.addOrchestratorVersion(dssOrchestratorVersion); + } catch (final Exception t) { + LOGGER.error("Failed to copy new orcVersion in orchestrator server for orc({}).", orcId, t); + DSSExceptionUtils.dealErrorException(60099, "Failed to copy app in orchestrator server", t, DSSOrchestratorErrorException.class); + } + return dssOrchestratorVersion.getId(); + } else { + throw new DSSOrchestratorErrorException(10023, "获取第三方应用的Ref为空，不能完成拷贝操作！"); + } + } +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ImportDSSOrchestratorPluginImpl.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ImportDSSOrchestratorPluginImpl.java new file mode 100644 index 000000000..0064c9471 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/impl/ImportDSSOrchestratorPluginImpl.java @@ -0,0 +1,194 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.impl; + +import static com.webank.wedatasphere.dss.orchestrator.publish.impl.ExportDSSOrchestratorPluginImpl.DEFAULT_ORC_NAME; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.common.utils.ZipHelper; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestProjectImportOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.AbstractDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorImportRequestRef; +import com.webank.wedatasphere.dss.orchestrator.core.service.BMLService; +import com.webank.wedatasphere.dss.orchestrator.core.utils.OrchestratorUtils; +import com.webank.wedatasphere.dss.orchestrator.db.dao.OrchestratorMapper; +import com.webank.wedatasphere.dss.orchestrator.loader.OrchestratorManager; +import com.webank.wedatasphere.dss.orchestrator.loader.utils.OrchestratorLoaderUtils; +import com.webank.wedatasphere.dss.orchestrator.publish.ImportDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.publish.io.input.MetaInputService; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.service.RefImportService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RefFactory; +import java.io.File; +import java.io.InputStream; +import java.util.Date; +import java.util.List; +import java.util.Map; +import javafx.util.Pair; +import org.apache.commons.lang.StringUtils; +import org.springframework.beans.BeanUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; +import org.springframework.transaction.annotation.Transactional; + + +@Component +public class ImportDSSOrchestratorPluginImpl extends AbstractDSSOrchestratorPlugin implements ImportDSSOrchestratorPlugin { + + @Autowired + private MetaInputService metaInputService; + @Autowired + private OrchestratorMapper orchestratorMapper; + @Autowired + private BMLService bmlService; + @Autowired + private ContextService contextService; + @Autowired + private OrchestratorManager orchestratorManager; + + @Override + @Transactional(rollbackFor = Exception.class) + public Long importOrchestrator(String userName, String workspaceName, String projectName, Long projectId, String resourceId, String version, + List dssLabels, Workspace workspace) throws Exception { + Long appId = null; + //1、下载BML的Orchestrator的导入包 + String inputZipPath = IoUtils.generateIOPath(userName, projectName, DEFAULT_ORC_NAME + ".zip"); + bmlService.downloadToLocalPath(userName, resourceId, version, inputZipPath); + String inputPath = ZipHelper.unzip(inputZipPath); + + IOEnv ioEnv = IoUtils.readIOEnv(inputPath); + //2、导入Info信息(导入冲突处理) + List dssOrchestratorInfos = metaInputService.importOrchestrator(inputPath); + DSSOrchestratorInfo importDssOrchestratorInfo = dssOrchestratorInfos.get(0); + + //根据工程ID和编排名称查询 + String uuid = orchestratorMapper.getOrcNameByParam(importDssOrchestratorInfo.getProjectId(),importDssOrchestratorInfo.getName()); + //通过UUID查找是否已经导入过 + DSSOrchestratorInfo existFlag = orchestratorMapper.getOrchestratorByUUID(importDssOrchestratorInfo.getUUID()); + //add 和update都需要更新成当前环境id信息，放到新的版本记录中 + //todo 跨环境必须保证工程ID一样，或者需要更新导入包中的所有工程ID为当前最新ID,不一致的话关系到上下文、第三方工程的映射问题 + if (null != existFlag) { + //判断是否存在相同名称的编排名称 + if (StringUtils.isNotBlank(uuid) && !uuid.equals(importDssOrchestratorInfo.getUUID())) { + DSSExceptionUtils + .dealErrorException(61002, "The same orchestration name already exists" ,DSSErrorException.class); + + } + //如果Orchestrator已经导入过，目前只更新版本信息，并更新基础信息name,其它信息不修改。 + orchestratorMapper.updateOrchestrator(importDssOrchestratorInfo); + } else { + //判断是否存在相同名称的编排名称 + if (StringUtils.isNotBlank(uuid)) { + DSSExceptionUtils + .dealErrorException(61002, "The same orchestration name already exists" ,DSSErrorException.class); + } + orchestratorMapper.addOrchestrator(importDssOrchestratorInfo); + } + String flowZipPath = inputPath + File.separator + "orc_flow.zip"; + //3、上传工作流zip包到bml + InputStream inputStream = bmlService.readLocalResourceFile(userName, flowZipPath); + Map resultMap = bmlService.upload(userName, inputStream, importDssOrchestratorInfo.getName() + "_orc_flow.zip", projectName); + String orcResourceId = resultMap.get("resourceId").toString(); + String orcBmlVersion = resultMap.get("version").toString(); + + + //4、导入版本Version信息 + DSSOrchestratorVersion dssOrchestratorVersion = new DSSOrchestratorVersion(); + dssOrchestratorVersion.setAppId(null); + dssOrchestratorVersion.setComment("orchestrator import"); + dssOrchestratorVersion.setOrchestratorId(importDssOrchestratorInfo.getId()); + dssOrchestratorVersion.setContent(""); + dssOrchestratorVersion.setProjectId(projectId); + dssOrchestratorVersion.setSource("Orchestrator create"); + dssOrchestratorVersion.setUpdater(userName); + + String oldVersion = orchestratorMapper.getLatestVersion(importDssOrchestratorInfo.getId()); + if (StringUtils.isNotEmpty(oldVersion)){ + dssOrchestratorVersion.setVersion(OrchestratorUtils.increaseVersion(oldVersion)); + }else{ + dssOrchestratorVersion.setVersion(OrchestratorUtils.generateNewVersion()); + } + + //5、生成上下文ContextId + String contextId = contextService.createContextID(workspaceName, projectName, importDssOrchestratorInfo.getName(), dssOrchestratorVersion.getVersion(), userName); + dssOrchestratorVersion.setFormatContextId(contextId); + LOGGER.info("Create a new ContextId for import: {} ", contextId); + + dssOrchestratorVersion.setUpdateTime(new Date()); + orchestratorMapper.addOrchestratorVersion(dssOrchestratorVersion); + + //6、导出第三方应用信息，如工作流、Visualis、Qualities + DSSOrchestrator dssOrchestrator = orchestratorManager.getOrCreateOrchestrator(userName, + workspaceName, importDssOrchestratorInfo.getType(), importDssOrchestratorInfo.getAppConnName(), dssLabels); + AppConn orchestratorAppConn = dssOrchestrator.getAppConn(); + Pair standardPair = OrchestratorLoaderUtils + .getOrcDevelopStandard(userName, workspaceName, importDssOrchestratorInfo, dssLabels); + if (null != standardPair) { + RefImportService refImportService = standardPair.getValue().getRefImportService(standardPair.getKey()); + OrchestratorImportRequestRef orchestratorImportRequestRef = null; + try { + orchestratorImportRequestRef = + (OrchestratorImportRequestRef) RefFactory.INSTANCE.newRef(OrchestratorImportRequestRef.class, + orchestratorAppConn.getClass().getClassLoader(), "com.webank.wedatasphere.dss.appconn.workflow.ref"); + } catch (final Exception e) { + DSSExceptionUtils + .dealErrorException(61001, "failed to new ref with class " + OrchestratorImportRequestRef.class.getName(), e, DSSErrorException.class); + } + orchestratorImportRequestRef.setResourceId(orcResourceId); + orchestratorImportRequestRef.setBmlVersion(orcBmlVersion); + orchestratorImportRequestRef.setProjectId(projectId); + orchestratorImportRequestRef.setUserName(userName); + orchestratorImportRequestRef.setWorkspaceName(workspaceName); + orchestratorImportRequestRef.setWorkspace(workspace); + orchestratorImportRequestRef.setProjectName(projectName); + orchestratorImportRequestRef.setContextID(contextId); + //todo getSourceEnv + orchestratorImportRequestRef.setSourceEnv(ioEnv); + orchestratorImportRequestRef.setOrcVersion(dssOrchestratorVersion.getVersion()); + CommonResponseRef responseRef = (CommonResponseRef) refImportService.getRefImportOperation().importRef(orchestratorImportRequestRef); + appId = responseRef.getOrcId(); + String appContent = responseRef.getContent(); + + //更新返回內容 + dssOrchestratorVersion.setAppId(appId); + dssOrchestratorVersion.setContent(appContent); + orchestratorMapper.updateOrchestratorVersion(dssOrchestratorVersion); + + } + + RequestProjectImportOrchestrator projectImportOrchestrator = new RequestProjectImportOrchestrator(); + BeanUtils.copyProperties(importDssOrchestratorInfo,projectImportOrchestrator); + projectImportOrchestrator.setVersionId(dssOrchestratorVersion.getId()); + //保存工程级别的编排模式 + DSSSenderServiceFactory.getOrCreateServiceInstance().getProjectServerSender() + .ask(projectImportOrchestrator); + return dssOrchestratorVersion.getOrchestratorId(); + } +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/MetaExportService.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/MetaExportService.java new file mode 100644 index 000000000..c35c817eb --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/MetaExportService.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.io.export; + + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; + +import java.io.IOException; + + +public interface MetaExportService { + + + /** + * + * @param dssOrchestratorInfo + * @param savePath + * @throws IOException + */ + void export(DSSOrchestratorInfo dssOrchestratorInfo, String savePath) throws IOException; + + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/MetaWriter.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/MetaWriter.java new file mode 100644 index 000000000..16b706e58 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/MetaWriter.java @@ -0,0 +1,172 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.io.export; + +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.publish.conf.DSSOrchestratorConf; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; + +public class MetaWriter { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + private final Class tClass; + private final String tableName; + private List datas = new ArrayList<>(); + private List ignoreFields = new ArrayList<>(); + private List fields = null; + private List comments = new ArrayList<>(); + private List table = new ArrayList<>(); + private String commentPrefix = "#"; + private String seperator = "|"; + + public MetaWriter(Class tClass, String tableName) { + this.tClass = tClass; + this.tableName = tableName; + } + + public static MetaWriter of(String tableName, Class tClass) { + return new MetaWriter<>(tClass, tableName); + } + + public MetaWriter ignore(String... fields) { + ignoreFields.addAll(Arrays.stream(fields).collect(Collectors.toList())); + return this; + } + + public MetaWriter data(List datas) { + this.datas.addAll(datas); + return this; + } + + public MetaWriter data(T data) { + this.datas.add(data); + return this; + } + + public MetaWriter comment(String... comments) { + this.comments.addAll(Arrays.stream(comments).collect(Collectors.toList())); + return this; + } + + /** + * 上传bml + * + * @return + */ + public InputStream write() { + writeComment(); + writeHead(); + writeBody(); + //table 添加换行符进行转流 + String tableStr = table.stream().reduce((a, b) -> a + "\n" + b).orElse("") + "\n"; + logger.info("\n" + tableStr); + return new ByteArrayInputStream(tableStr.getBytes()); + } + + /** + * 写入本地 + * + * @param outputStream + */ + public void write(OutputStream outputStream) throws IOException { + writeComment(); + writeHead(); + writeBody(); + IOUtils.writeLines(table, "\n", outputStream); + } + + private void writeBody() { + datas.forEach(DSSExceptionUtils.handling(this::writeBody)); + } + + private void writeBody(T t) throws NoSuchFieldException, IllegalAccessException { + ArrayList line = new ArrayList<>(); + for (String field : fields) { + Field declaredField = tClass.getDeclaredField(field); + declaredField.setAccessible(true); + Object o = declaredField.get(t); + if (o != null) { + line.add(o.toString()); + } else { + line.add(null); + } + } + table.add(reduce(line)); + } + + private String reduce(List strs) { + return strs.stream().reduce((a, b) -> a + seperator + b).orElse(""); + } + + /** + * 写表的头部 + * 1.过滤ignore的属性 + * 2.驼峰转mysql的 _ (暂时略过) + * 3.写入List table + */ + private void writeHead() { + fields = Arrays.stream(tClass.getDeclaredFields()) + .map(Field::getName) + .filter(n -> !ignoreFields.contains(n)) + .collect(Collectors.toList()); + table.add(reduce(fields)); + } + + /** + * 驼峰转_ + * + * @param str + * @return + */ + private String unCamel(String str) { + // TODO: 2020/3/9 + return null; + } + + /** + * 写comment,包括表名,class名,和外部自定义的comment + */ + private void writeComment() { + table.add(connectCommentPrefix(String.format("tableName:%s", tableName))); + table.add(connectCommentPrefix(String.format("class:%s", tClass.getName()))); + table.add(connectCommentPrefix(String.format("env:%s", DSSOrchestratorConf.DSS_EXPORT_ENV.getValue()))); + comments.stream().map(this::connectCommentPrefix).forEach(table::add); + } + + /** + * str前加上comment标识 + * + * @param str + * @return + */ + private String connectCommentPrefix(String str) { + return commentPrefix + str; + } + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/impl/MetaExportServiceImpl.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/impl/MetaExportServiceImpl.java new file mode 100644 index 000000000..929148faf --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/export/impl/MetaExportServiceImpl.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.io.export.impl; + + +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.publish.io.export.MetaExportService; +import com.webank.wedatasphere.dss.orchestrator.publish.io.export.MetaWriter; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; +import java.io.*; + + +@Service +public class MetaExportServiceImpl implements MetaExportService { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + + private final String fileName = "meta.txt"; + + + @Override + public void export(DSSOrchestratorInfo dssOrchestratorInfo, String savePath) throws IOException { + + try ( + OutputStream outputStream = generateOutputStream(savePath) + ) { + exportOrchestratorBaseInfo(dssOrchestratorInfo, outputStream); + } + } + + + private OutputStream generateOutputStream(String basePath) throws IOException { + return IoUtils.generateExportOutputStream(basePath + File.separator + fileName); + } + + + private void exportOrchestratorBaseInfo(DSSOrchestratorInfo dssOrchestratorInfo,OutputStream outputStream) throws IOException { + + MetaWriter.of("dss_orchestrator", DSSOrchestratorInfo.class).data(dssOrchestratorInfo).write(outputStream); + + } + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/MetaInputService.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/MetaInputService.java new file mode 100644 index 000000000..230ad50b7 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/MetaInputService.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.io.input; + + + + + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; + +import java.io.IOException; +import java.util.List; + +public interface MetaInputService { + + + + List importOrchestrator(String basePath) throws IOException; + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/MetaReader.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/MetaReader.java new file mode 100644 index 000000000..7b0854106 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/MetaReader.java @@ -0,0 +1,166 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.io.input; + +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.lang.reflect.Field; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Locale; +import java.util.stream.Collectors; + +public class MetaReader { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + private final Class tClass; + private final String tableName; + private String commentPrefix = "#"; + private String seperator = "\\|"; + private List datas = new ArrayList<>(); + private List fields = null; + private List comments = new ArrayList<>(); + private List> body = new ArrayList<>(); + private boolean finded = false; + private boolean firstLine = true; + + public MetaReader(Class tClass, String tableName) { + this.tClass = tClass; + this.tableName = tableName; + } + + public static MetaReader of(String tableName, Class tClass) { + return new MetaReader<>(tClass, tableName); + } + + public List read(InputStream inputStream) throws IOException { + try (InputStreamReader streamReader = new InputStreamReader(inputStream); + BufferedReader reader = new BufferedReader(streamReader);) { + readTable(reader); + } + readT(); + return datas; + } + + public String read(InputStream inputStream, String key) throws IOException { + try (InputStreamReader streamReader = new InputStreamReader(inputStream); + BufferedReader reader = new BufferedReader(streamReader);) { + readTable(reader); + } + String comment = comments.stream().filter(c -> c.contains(key + ":")).findFirst().orElse(""); + String[] split = comment.split(":"); + if (split.length > 1) return split[1]; + return ""; + } + + private void readT() { + body.stream().map(DSSExceptionUtils.map(this::lineToT)).forEach(datas::add); + } + + private T lineToT(List list) throws IllegalAccessException, InstantiationException, NoSuchFieldException, ParseException { + T t = tClass.newInstance(); + for (int i = 0; i < list.size(); i++) { + String valueStr = list.get(i); + if ("null".equalsIgnoreCase(valueStr)) continue; + Field declaredField = tClass.getDeclaredField(fields.get(i)); + declaredField.setAccessible(true); + Object value = null; + String type = declaredField.getType().getSimpleName(); + switch (type) { + case "String": + value = valueStr; + break; + case "Date": + value = new SimpleDateFormat("EEE MMM dd HH:mm:ss ZZZ yyyy", Locale.ENGLISH).parse(valueStr); + break; + case "Long": + case "long": + value = Long.valueOf(valueStr); + break; + case "Boolean": + value = Boolean.valueOf(valueStr); + break; + case "Integer": + value = Integer.valueOf(valueStr); + break; + default: + logger.warn(String.format("unsupport type %s", type)); + } + declaredField.set(t, value); + } + return t; + } + + private void readTable(BufferedReader reader) throws IOException { + String line = null; + while ((line = reader.readLine()) != null) { + if (!finded && !shut(line)) { + continue; + } + if (finded && isTableName(line)) { + break; + } + if (shut(line)) { + finded = true; + continue; + } + // TODO: 2020/3/9 + if (isComment(line)) { + comments.add(line); + continue; + } + if (firstLine) { + //handle head + fields = Arrays.stream(line.split(seperator)).collect(Collectors.toList()); + firstLine = false; + continue; + } + body.add(Arrays.stream(line.split(seperator)).collect(Collectors.toList())); + //handle body + } + } + + private boolean isComment(String str) { + return str.startsWith(commentPrefix); + } + + private boolean isTableName(String str) { + return isComment(str) && str.contains(commentPrefix + "tableName:"); + } + + private boolean isClass(String str) { + return isComment(str) && str.contains(commentPrefix + "class:"); + } + + private String getComment(String str) { + return str.substring(1); + } + + private boolean shut(String str) { + return isTableName(str) && getComment(str).equals(String.format("tableName:%s", tableName)); + } +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/impl/MetaInputServiceImpl.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/impl/MetaInputServiceImpl.java new file mode 100644 index 000000000..52b9babbc --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/io/input/impl/MetaInputServiceImpl.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.io.input.impl; + + +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.publish.io.input.MetaInputService; +import com.webank.wedatasphere.dss.orchestrator.publish.io.input.MetaReader; + +import org.springframework.stereotype.Service; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.util.List; + + +@Service +public class MetaInputServiceImpl implements MetaInputService { + + // TODO: 2020/3/13 防止表结构发生改变的version 字段的添加 + + private final String fileName = "meta.txt"; + + @Override + public List importOrchestrator(String basePath) throws IOException { + try (InputStream inputStream = generateInputstream(basePath)) { + return MetaReader.of("dss_orchestrator", DSSOrchestratorInfo.class).read(inputStream); + } + } + + + /** + * 获取inputStream + * + * @param basePath + * @return + * @throws FileNotFoundException + */ + private InputStream generateInputstream(String basePath) throws IOException { + return IoUtils.generateInputInputStream(basePath + File.separator + fileName); + } + + +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/job/ConversionJobEntity.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/job/ConversionJobEntity.java new file mode 100644 index 000000000..cef04dd32 --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/job/ConversionJobEntity.java @@ -0,0 +1,119 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.job; + +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import java.util.Date; +import java.util.List; + + +public class ConversionJobEntity { + + private DSSProject project; + + private List orcIdList; + + private List refAppIdList; + + private Workspace workspace; + + private String userName; + + private List labels; + + private Date createTime; + + private Date updateTime; + + private ResponseOperateOrchestrator response; + + public DSSProject getProject() { + return project; + } + + public void setProject(DSSProject project) { + this.project = project; + } + + public List getOrcIdList() { + return orcIdList; + } + + public void setOrcIdList(List orcIdList) { + this.orcIdList = orcIdList; + } + + public List getRefAppIdList() { + return refAppIdList; + } + + public void setRefAppIdList(List refAppIdList) { + this.refAppIdList = refAppIdList; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public ResponseOperateOrchestrator getResponse() { + return response; + } + + public void setResponse(ResponseOperateOrchestrator response) { + updateTime = new Date(); + this.response = response; + } + + public List getLabels() { + return labels; + } + + public void setLabels(List labels) { + this.labels = labels; + } + + public Workspace getWorkspace() { + return workspace; + } + + public void setWorkspace(Workspace workspace) { + this.workspace = workspace; + } +} diff --git a/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/job/OrchestratorConversionJob.java b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/job/OrchestratorConversionJob.java new file mode 100644 index 000000000..3e53f411c --- /dev/null +++ b/dss-framework/framework-plugins/dss-framework-orchestrator-publish/src/main/java/com/webank/wedatasphere/dss/orchestrator/publish/job/OrchestratorConversionJob.java @@ -0,0 +1,107 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.publish.job; + +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.protocol.project.ProjectInfoRequest; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.orchestrator.publish.ConversionDSSOrchestratorPlugin; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.linkis.common.utils.ByteTimeUtils; +import java.util.List; +import java.util.function.Consumer; +import org.apache.commons.lang.exception.ExceptionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public final class OrchestratorConversionJob implements Runnable { + + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorConversionJob.class); + + private String id; + + private ConversionJobEntity conversionJobEntity; + private List conversionDSSOrchestratorPlugins; + private Consumer consumer; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public void setConversionJobEntity(ConversionJobEntity conversionJobEntity) { + this.conversionJobEntity = conversionJobEntity; + } + + public ConversionJobEntity getConversionJobEntity() { + return conversionJobEntity; + } + + public void setConversionDSSOrchestratorPlugins(List conversionDSSOrchestratorPlugins) { + this.conversionDSSOrchestratorPlugins = conversionDSSOrchestratorPlugins; + } + + public void afterConversion(Consumer consumer) { + this.consumer = consumer; + } + + @Override + public void run() { + //1.从编排中心导出一次工作流,进行一次版本升级 + //2.进行发布到schedulis等调度系统 + LOGGER.info("Begin to convert project {} for user {} to scheduler, the orcIdList is {}.", + conversionJobEntity.getProject().getId(), conversionJobEntity.getUserName(), conversionJobEntity.getOrcIdList()); + conversionJobEntity.setResponse(ResponseOperateOrchestrator.running()); + ConversionDSSOrchestratorPlugin conversionDSSOrchestratorPlugin = null; + for (DSSOrchestratorPlugin plugin: conversionDSSOrchestratorPlugins) { + if(plugin instanceof ConversionDSSOrchestratorPlugin) { + conversionDSSOrchestratorPlugin = (ConversionDSSOrchestratorPlugin) plugin; + } + } + ProjectInfoRequest projectInfoRequest = new ProjectInfoRequest(); + projectInfoRequest.setProjectId(conversionJobEntity.getProject().getId()); + try{ + DSSProject project = (DSSProject) DSSSenderServiceFactory.getOrCreateServiceInstance().getProjectServerSender() + .ask(projectInfoRequest); + conversionJobEntity.setProject(project); + Workspace workspace = conversionJobEntity.getWorkspace(); + ResponseOperateOrchestrator response = conversionDSSOrchestratorPlugin.convert(conversionJobEntity.getUserName(), project, workspace, + conversionJobEntity.getRefAppIdList(), conversionJobEntity.getLabels()); + if(response.isFailed()) { + String msg = response.getMessage() == null ? "Unknown reason, please ask admin for help!" : response.getMessage(); + throw new DSSErrorException(50000, msg); + } + //3.如果都没有报错，那么默认任务应该是成功的,那么则将所有的状态进行置为完成 + consumer.accept(response); + conversionJobEntity.setResponse(response); + } catch (final Exception t){ + LOGGER.error("Convert for project {} failed.", conversionJobEntity.getProject(), t); + ResponseOperateOrchestrator response = ResponseOperateOrchestrator.failed(ExceptionUtils.getRootCauseMessage(t)); + conversionJobEntity.setResponse(response); + consumer.accept(response); + } + LOGGER.info("convert project {} for user {} to scheduler {}, costs {}.", conversionJobEntity.getResponse().getJobStatus(), conversionJobEntity.getProject().getId(), + conversionJobEntity.getUserName(), ByteTimeUtils.msDurationToString(conversionJobEntity.getUpdateTime().getTime() - conversionJobEntity.getCreateTime().getTime())); + } +} diff --git a/dss-framework/pom.xml b/dss-framework/pom.xml new file mode 100644 index 000000000..7a1d8c2ce --- /dev/null +++ b/dss-framework/pom.xml @@ -0,0 +1,40 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-framework + pom + + + dss-framework-common + dss-appconn-framework + dss-framework-workspace-server + framework-plugins/dss-framework-orchestrator-publish + dss-framework-orchestrator-server + dss-framework-project-server + + + \ No newline at end of file diff --git a/dss-linkis-node-execution/pom.xml b/dss-linkis-node-execution/pom.xml deleted file mode 100644 index f773b12f2..000000000 --- a/dss-linkis-node-execution/pom.xml +++ /dev/null @@ -1,69 +0,0 @@ - - - - - 4.0.0 - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - - dss-linkis-node-execution - - - - com.webank.wedatasphere.linkis - linkis-ujes-client - ${linkis.version} - - - - com.webank.wedatasphere.linkis - linkis-workspace-httpclient - ${linkis.version} - - - - - - - - - - org.apache.maven.plugins - maven-compiler-plugin - 3.2 - - 1.8 - 1.8 - UTF-8 - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - - \ No newline at end of file diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContext.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContext.java deleted file mode 100644 index a10b6962a..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContext.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution; - -import java.util.Iterator; -import java.util.List; -import java.util.Map; - -/** - * Created by enjoyyin on 2019/10/29. - */ -public interface WorkflowContext { - - Object getValue(String key); - - List getValues(String prefix); - - Iterator keyIterator(); - - Map getSubMapByPrefix(String keyPrefix); - - - void setValue(String key, Object value, int readTimes); - - static WorkflowContext getAppJointContext() { - return WorkflowContextFactory.getWorkflowContext(); - } - -} \ No newline at end of file diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContextFactory.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContextFactory.java deleted file mode 100644 index fdc2ed1e7..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContextFactory.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution; - -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionWarnException; -import org.apache.commons.lang.ClassUtils; -import org.apache.commons.lang.StringUtils; - -/** - * Created by enjoyyin on 2019/10/29. - */ -public class WorkflowContextFactory { - - private static Class clazz = WorkflowContextImpl.class; - private static WorkflowContext workflowContext = null; - - public static void register(Class clazz) { - WorkflowContextFactory.clazz = clazz; - } - - static WorkflowContext getWorkflowContext() { - if(workflowContext == null) { - synchronized (WorkflowContextFactory.class) { - if(workflowContext == null) { - String className = LinkisJobExecutionConfiguration.APPJOINT_CONTEXT_CLASS.acquireNew(); - if(clazz == WorkflowContextImpl.class && StringUtils.isNotBlank(className)) { - try { - clazz = ClassUtils.getClass(className); - } catch (ClassNotFoundException e) { - throw new LinkisJobExecutionWarnException(90150, "find class + " + className + " failed!", e); - } - } - try { - workflowContext = clazz.newInstance(); - } catch (Exception e) { - throw new LinkisJobExecutionWarnException(90150, "initial class + " + className + " failed!", e); - } - } - } - } - return workflowContext; - } -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContextImpl.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContextImpl.java deleted file mode 100644 index 8f2ff84fa..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/WorkflowContextImpl.java +++ /dev/null @@ -1,96 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution; - -import com.google.common.cache.Cache; -import com.google.common.cache.CacheBuilder; -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.entity.ContextInfo; -import org.apache.commons.lang.StringUtils; - -import java.util.*; -import java.util.concurrent.TimeUnit; - -/** - * Created by johnnwang on 2019/9/26. - */ -public class WorkflowContextImpl implements WorkflowContext { - - private Cache contextCache = CacheBuilder.newBuilder() - .maximumSize(LinkisJobExecutionConfiguration.LINKIS_CACHE_MAX_SIZE.getValue()) - .expireAfterWrite(LinkisJobExecutionConfiguration.LINKIS_CACHE_EXPIRE_TIME.getValue(), TimeUnit.HOURS) - .build(); - - - @Override - public Object getValue(String key) { - ContextInfo contextInfo = contextCache.getIfPresent(key); - if (contextInfo != null && contextInfo.getReadNum() > 0) { - int readNum = contextInfo.getReadNum() - 1; - Object value = contextInfo.getValue(); - if (readNum <= 0) { - contextCache.invalidate(key); - } - contextInfo.setReadNum(contextInfo.getReadNum() - 1); - return value; - } - if (contextInfo != null && contextInfo.getReadNum() < 0) { - return contextInfo.getValue(); - } - return null; - } - - @Override - public List getValues(String prefix) { - List values = new ArrayList<>(); - Iterator keys = keyIterator(); - while (keys.hasNext()) { - String key = keys.next(); - if (key.startsWith(prefix)) { - values.add(getValue(key)); - } - } - return values; - } - - @Override - public Iterator keyIterator() { - return contextCache.asMap().keySet().iterator(); - } - - @Override - public Map getSubMapByPrefix(String keyPrefix) { - Map map = new HashMap<>(); - Iterator keys = keyIterator(); - while (keys.hasNext()) { - String key = keys.next(); - if (key.startsWith(keyPrefix)) { - map.put(StringUtils.substringAfter(key, keyPrefix), getValue(key)); - } - } - return map; - } - - @Override - public void setValue(String key, Object value, int readNum) { - - contextCache.put(key, new ContextInfo(value, readNum)); - - } - -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/conf/LinkisJobExecutionConfiguration.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/conf/LinkisJobExecutionConfiguration.java deleted file mode 100644 index 9b75d73a0..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/conf/LinkisJobExecutionConfiguration.java +++ /dev/null @@ -1,101 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.conf; - - - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; - -/** - * Created by peacewong on 2019/9/19. - */ -public class LinkisJobExecutionConfiguration { - - - public static final String LINKIS_TYPE = "linkistype"; - - public static final String APPJOINT = "appjoint"; - - - - - - - - - public static final String PROJECT_PREFIX = "project"; - public static final String FLOW_PREFIX = "flow"; - public static final String JOB_PREFIX = "job"; - - public static final String FLOW_VARIABLE_PREFIX = "flow.variable."; - - public static final String NODE_CONF_PREFIX = "node.conf."; - - public static final String CONF_STARTUP = "startup"; - public static final String CONF_RUNTIME = "runtime"; - public static final String CONF_SPECIAL = "special"; - - public static final String RESOURCES_NAME = ".resources"; - - - public static final String WORKFLOW_SHARED_NODES_JOBIDS = "workflow.shared.nodes.jobids"; - - public static final String LINKIS_SUBMIT_USER = "wds.linkis.schedulis.submit.user"; - - public static final String LINKIS_CONTROL_EMPTY_NODE = "linkis.control.empty"; - - public static final String LINKIS_DATA_EXCHANGE_NODE = "linkis.data.exchange"; - - public final static CommonVars JOB_DEFAULT_TYPE = CommonVars.apply("wds.linkis.flow.default.job.type","linkis"); - - public final static CommonVars LINKIS_DEFAULT_TYPE = CommonVars.apply("wds.linkis.flow.default.linkis.type","spark.sql"); - - - public final static CommonVars LINKIS_CONNECTION_TIMEOUT = CommonVars.apply("wds.linkis.flow.connection.timeout",30000); - - public final static CommonVars LINKIS_JOB_REQUEST_STATUS_TIME = CommonVars.apply("wds.linkis.flow.connection.timeout",3000); - - public final static CommonVars LINKIS_ADMIN_USER = CommonVars.apply("wds.linkis.client.flow.adminuser","ws"); - - - - - - public final static CommonVars LINKIS_AUTHOR_USER_TOKEN = CommonVars.apply("wds.linkis.client.flow.author.user.token","WS-AUTH"); - - public final static CommonVars LINKIS_JOB_CREATOR = CommonVars.apply("wds.linkis.flow.job.creator","nodeexecution"); - - public final static CommonVars LINKIS_URL = CommonVars.apply("wds.linkis.gateway.url","wtss"); - - public final static CommonVars LINKIS_API_VERSION = CommonVars.apply("wds.linkis.server.version","v1"); - - public final static CommonVars LINKIS_CACHE_MAX_SIZE = CommonVars.apply("wds.linkis.context.cache.max.size",100000); - - public final static CommonVars LINKIS_CACHE_EXPIRE_TIME = CommonVars.apply("wds.linkis.context.cache.expire.time",24); - - public final static CommonVars RESULT_PRINT_SIZE = CommonVars.apply("wds.linkis.result.print.size",10); - - public final static CommonVars APPJOINT_CONTEXT_CLASS = CommonVars.apply("wds.dss.appjoint.context",""); - - - public final static CommonVars LOG_SIZE = CommonVars.apply("wds.linkis.log.size",-1); - public final static CommonVars LOG_ARRAY_LEN = CommonVars.apply("wds.linkis.log.array.len",4); - - public final static String SUCCEED= "Succeed"; - -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/impl/LinkisNodeExecutionImpl.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/impl/LinkisNodeExecutionImpl.java deleted file mode 100644 index 264e8288c..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/impl/LinkisNodeExecutionImpl.java +++ /dev/null @@ -1,203 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.execution.impl; - -import com.webank.wedatasphere.dss.linkis.node.execution.WorkflowContext; -import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; -import com.webank.wedatasphere.dss.linkis.node.execution.execution.LinkisNodeExecution; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SharedJob; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SignalSharedJob; -import com.webank.wedatasphere.dss.linkis.node.execution.listener.LinkisExecutionListener; -import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobRuntimeParamsParser; -import com.webank.wedatasphere.dss.linkis.node.execution.service.impl.BuildJobActionImpl; -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.dss.linkis.node.execution.parser.CodeParser; -import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParser; -import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisUjesClientUtils; -import com.webank.wedatasphere.linkis.common.utils.Utils; -import com.webank.wedatasphere.linkis.ujes.client.UJESClient; -import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; -import com.webank.wedatasphere.linkis.ujes.client.request.ResultSetAction; -import com.webank.wedatasphere.linkis.ujes.client.response.JobInfoResult; -import com.webank.wedatasphere.linkis.ujes.client.response.JobLogResult; -import org.apache.commons.lang.StringUtils; - -import java.util.ArrayList; -import java.util.Map; - -/** - * Created by peacewong on 2019/11/2. - */ -public class LinkisNodeExecutionImpl implements LinkisNodeExecution , LinkisExecutionListener { - - private static LinkisNodeExecution linkisExecution = new LinkisNodeExecutionImpl(); - - private LinkisNodeExecutionImpl() { - registerJobParser(new CodeParser()); - /*registerJobParser(new JobParamsParser());*/ - registerJobParser(new JobRuntimeParamsParser()); - } - - - public static LinkisNodeExecution getLinkisNodeExecution() { - return linkisExecution; - } - - private UJESClient client; - - private ArrayList jobParsers = new ArrayList<>(); - - public UJESClient getClient(Map props) { - if (client == null) { - synchronized (LinkisNodeExecution.class) { - if (client == null) { - client = LinkisUjesClientUtils.getUJESClient( - LinkisJobExecutionConfiguration.LINKIS_URL.getValue(props), - LinkisJobExecutionConfiguration.LINKIS_ADMIN_USER.getValue(props), - LinkisJobExecutionConfiguration.LINKIS_AUTHOR_USER_TOKEN.getValue(props), - props); - } - } - } - return client; - } - - public void registerJobParser(JobParser jobParser) { - this.jobParsers.add(jobParser); - } - - @Override - public void runJob(Job job) throws Exception { - // job parser - for(JobParser parser : jobParsers){ - parser.parseJob(job); - } - JobExecuteAction jobAction = BuildJobActionImpl.getbuildJobAction().getJobAction(job); - job.setJobExecuteResult(getClient(job.getJobProps()).execute(jobAction)); - job.getLogObj().info("<---------------Start to execute job--------------->"); - } - - @Override - public String getState(Job job) { - return getClient(job.getJobProps()).getJobInfo(job.getJobExecuteResult()).getJobStatus(); - } - - @Override - public String getLog(Job job) { - - JobLogResult jobLogResult = getClient(job.getJobProps()) - .log(job.getJobExecuteResult(), - job.getLogFromLine(), - LinkisJobExecutionConfiguration.LOG_SIZE.getValue()); - - job.setLogFromLine(jobLogResult.fromLine()); - - ArrayList logArray = jobLogResult.getLog(); - - if (logArray != null && logArray.size() - >= LinkisJobExecutionConfiguration.LOG_ARRAY_LEN.getValue() - && StringUtils.isNotEmpty(logArray.get(3))) { - return logArray.get(3); - } - return null; - } - - @Override - public void waitForComplete(Job job) throws Exception { - JobInfoResult jobInfo = getClient(job.getJobProps()).getJobInfo(job.getJobExecuteResult()); - while (!jobInfo.isCompleted()) { - job.getLogObj().info("Update Progress info:" + this.getProgress(job)); - job.getLogObj().info("<----linkis log ---->"); - String log = this.getLog(job); - if (log != null) { - job.getLogObj().info(log); - } - job.getLogObj().info("<----linkis log ---->"); - Utils.sleepQuietly(LinkisJobExecutionConfiguration.LINKIS_JOB_REQUEST_STATUS_TIME.getValue(job.getJobProps())); - jobInfo = getClient(job.getJobProps()).getJobInfo(job.getJobExecuteResult()); - } - if (!jobInfo.isSucceed()) { - throw new LinkisJobExecutionErrorException(90101, "Failed to execute Job: " + jobInfo.getMessage()); - } - } - - @Override - public void cancel(Job job) throws Exception { - getClient(job.getJobProps()).kill(job.getJobExecuteResult()); - } - - @Override - public double getProgress(Job job) { - return getClient(job.getJobProps()).progress(job.getJobExecuteResult()).getProgress(); - } - - @Override - public Boolean isCompleted(Job job) { - return getClient(job.getJobProps()).getJobInfo(job.getJobExecuteResult()).isCompleted(); - } - - @Override - public int getResultSize(Job job) { - JobInfoResult jobInfo = getClient(job.getJobProps()).getJobInfo(job.getJobExecuteResult()); - if (jobInfo.isSucceed()) { - String[] resultSetList = jobInfo.getResultSetList(getClient(job.getJobProps())); - if (resultSetList != null && resultSetList.length > 0) { - return resultSetList.length; - } - } - return 0; - } - - @Override - public String getResult(Job job, int index, int maxSize) { - String resultContent = null; - JobInfoResult jobInfo = getClient(job.getJobProps()).getJobInfo(job.getJobExecuteResult()); - String[] resultSetList = jobInfo.getResultSetList(getClient(job.getJobProps())); - if (resultSetList != null && resultSetList.length > 0) { - Object fileContent = client.resultSet(ResultSetAction.builder() - .setPath(resultSetList[index]) - .setUser(job.getJobExecuteResult().getUser()) - .setPageSize(maxSize).build()).getFileContent(); - if (fileContent instanceof ArrayList) { - ArrayList> resultSetRow = (ArrayList>) fileContent; - resultContent = StringUtils.join(resultSetRow, "\n"); - } else { - resultContent = fileContent.toString(); - } - } - return resultContent; - } - - @Override - public void onStatusChanged(String fromState, String toState, Job job) { - if (LinkisJobExecutionConfiguration.SUCCEED.equalsIgnoreCase(toState)){ - if (job instanceof SignalSharedJob){ - SignalSharedJob signalSharedJob = (SignalSharedJob) job; - String result = getResult(job, 0, -1); - WorkflowContext.getAppJointContext().setValue(signalSharedJob.getSharedKey(), result , -1); - } else if(job instanceof SharedJob){ - String taskId = job.getJobExecuteResult().getTaskID(); - job.getLogObj().info("Set shared info:" + taskId); - SharedJob sharedJob = ((SharedJob)job); - WorkflowContext.getAppJointContext().setValue(sharedJob.getSharedKey(), taskId , sharedJob.getSharedNum()); - } - - } - } -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractAppJointLinkisJob.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractAppJointLinkisJob.java deleted file mode 100644 index 7d9e47f06..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractAppJointLinkisJob.java +++ /dev/null @@ -1,198 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -import com.webank.wedatasphere.dss.linkis.node.execution.log.LinkisJobExecutionLog; -import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; - -import java.util.Map; - -/** - * Created by peacewong on 2019/11/3. - */ -public abstract class AbstractAppJointLinkisJob extends AppJointLinkisJob { - - private Map jobProps; - - - - private Map source; - - private JobTypeEnum jobType; - - private Map variables; - - private Map configuration; - - private String code; - - private String engineType; - - private String runType; - - - - private Map params; - - private Map runtimeParams; - - private JobExecuteResult jobExecuteResult; - - private LinkisJobExecutionLog logObj; - - private int logFromLine; - - - - - - - @Override - public Map getSource() { - return this.source; - } - - @Override - public JobTypeEnum getJobType() { - return this.jobType; - } - - - @Override - public Map getVariables() { - return this.variables; - } - - @Override - public Map getConfiguration() { - return this.configuration; - } - - @Override - public String getCode() { - return this.code; - } - - @Override - public void setCode(String code) { - this.code = code; - } - - @Override - public String getEngineType() { - return this.engineType; - } - - @Override - public String getRunType() { - return this.runType; - } - - - - @Override - public Map getParams() { - return this.params; - } - - @Override - public Map getRuntimeParams() { - return this.runtimeParams; - } - - @Override - public JobExecuteResult getJobExecuteResult() { - return this.jobExecuteResult; - } - - @Override - public void setJobExecuteResult(JobExecuteResult jobExecuteResult) { - this.jobExecuteResult = jobExecuteResult; - } - - @Override - public Map getJobProps() { - return this.jobProps; - } - - @Override - public void setJobProps(Map jobProps) { - this.jobProps = jobProps; - } - - @Override - public LinkisJobExecutionLog getLogObj() { - return this.logObj; - } - - @Override - public int getLogFromLine() { - return this.logFromLine; - } - - @Override - public void setLogFromLine(int index) { - this.logFromLine = index; - } - - - @Override - public void setSource(Map source) { - this.source = source; - } - - @Override - public void setJobType(JobTypeEnum jobType) { - this.jobType = jobType; - } - - @Override - public void setVariables(Map variables) { - this.variables = variables; - } - - @Override - public void setConfiguration(Map configuration) { - this.configuration = configuration; - } - - @Override - public void setEngineType(String engineType) { - this.engineType = engineType; - } - - @Override - public void setRunType(String runType) { - this.runType = runType; - } - - @Override - public void setParams(Map params) { - this.params = params; - } - - @Override - public void setRuntimeParams(Map runtimeParams) { - this.runtimeParams = runtimeParams; - } - - @Override - public void setLogObj(LinkisJobExecutionLog logObj) { - this.logObj = logObj; - } -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AppJointLinkisJob.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AppJointLinkisJob.java deleted file mode 100644 index 2b12e40cf..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AppJointLinkisJob.java +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -/** - * Created by peacewong on 2019/11/2. - */ -public abstract class AppJointLinkisJob extends LinkisJob { -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Builder.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Builder.java deleted file mode 100644 index 6409b9f37..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Builder.java +++ /dev/null @@ -1,150 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -import com.webank.wedatasphere.dss.linkis.node.execution.WorkflowContext; -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; -import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils; -import org.apache.commons.lang.StringUtils; - -import java.util.HashMap; -import java.util.Map; - -/** - * Created by peacewong on 2019/11/13. - */ -public abstract class Builder { - - protected abstract String getJobType(); - - - protected abstract Boolean isReadNode(); - - protected abstract Boolean isShareNod(); - - protected abstract Boolean isSignalSharedNode(); - - protected abstract ReadJob createReadJob(boolean isLinkisType); - - protected abstract SharedJob createSharedJob(boolean isLinkisType); - - protected abstract SignalSharedJob createSignalSharedJob(boolean isLinkisType); - - - protected abstract LinkisJob creatLinkisJob(boolean isLinkisType); - - protected abstract void fillJobInfo(Job job); - - protected abstract void fillLinkisJobInfo(LinkisJob linkisJob); - - protected abstract void fillCommonLinkisJobInfo(CommonLinkisJob linkisAppjointJob); - - - public Job build() throws Exception { - - LinkisJob job = null; - String jobType = getJobType(); - String[] jobTypeSplit = jobType.split("\\."); - if (jobTypeSplit.length < 3) { - throw new LinkisJobExecutionErrorException(90100, "This is not Linkis job type,this jobtype is " + jobType); - } - String engineType = jobTypeSplit[1]; - //delete linkis.engineType - String runType = StringUtils.substringAfterLast(jobType, jobTypeSplit[0] + "." + jobTypeSplit[1] + "."); - - if (LinkisJobExecutionConfiguration.LINKIS_CONTROL_EMPTY_NODE.equalsIgnoreCase(jobType)) { - job = new AbstractCommonLinkisJob() { - @Override - public String getSubmitUser() { - return null; - } - - @Override - public String getUser() { - return null; - } - - @Override - public String getJobName() { - return null; - } - }; - - job.setJobType(JobTypeEnum.EmptyJob); - return job; - } - if (LinkisJobExecutionUtils.isCommonAppjointJob(engineType)) { - if (isReadNode()) { - ReadJob readJob = createReadJob(false); - readJob.setSharedNodesInfo(getSharedNodesAndJobId(readJob)); - job = (LinkisJob) readJob; - job.setJobType(JobTypeEnum.ReadJob); - } else if (isShareNod()) { - SharedJob sharedJob = createSharedJob(false); - job = (LinkisJob) sharedJob; - job.setJobType(JobTypeEnum.ShareJob); - } else if (isSignalSharedNode()) { - job = (LinkisJob) createSignalSharedJob(false); - job.setJobType(JobTypeEnum.ShareJob); - } else { - job = creatLinkisJob(false); - job.setJobType(JobTypeEnum.CommonJob); - } - } else { - if (isReadNode()) { - ReadJob readJob = createReadJob(true); - readJob.setSharedNodesInfo(getSharedNodesAndJobId(readJob)); - job = (LinkisJob) readJob; - job.setJobType(JobTypeEnum.ReadJob); - } else if (isShareNod()) { - SharedJob sharedJob = createSharedJob(true); - job = (LinkisJob) sharedJob; - job.setJobType(JobTypeEnum.ShareJob); - }else if (isSignalSharedNode()) { - job = (LinkisJob) createSignalSharedJob(true); - job.setJobType(JobTypeEnum.ShareJob); - } else { - job = creatLinkisJob(true); - job.setJobType(JobTypeEnum.CommonJob); - } - fillCommonLinkisJobInfo((CommonLinkisJob) job); - } - - job.setEngineType(engineType); - job.setRunType(runType); - fillJobInfo(job); - fillLinkisJobInfo(job); - - return job; - } - - - private Map getSharedNodesAndJobId(ReadJob job) { - Map map = new HashMap<>(); - String[] sharedIds = job.getShareNodeIds(); - if (sharedIds == null) { - return map; - } - for (String nodeId : sharedIds) { - map.put(nodeId, WorkflowContext.getAppJointContext().getValue(job.getSharedKey(nodeId))); - } - return map; - } - -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobSignalKeyCreator.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobSignalKeyCreator.java deleted file mode 100644 index ce64bf901..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobSignalKeyCreator.java +++ /dev/null @@ -1,25 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -public interface JobSignalKeyCreator { - - String getSignalKeyByJob(Job job); - - String getSignalKeyBySignalSharedJob(SignalSharedJob job); -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/LinkisJob.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/LinkisJob.java deleted file mode 100644 index b2b6f1141..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/LinkisJob.java +++ /dev/null @@ -1,46 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - - -import java.util.Map; - -/** - * Created by peacewong on 2019/10/31. - */ -public abstract class LinkisJob implements Job{ - - - public abstract Map getSource(); - - public abstract void setSource(Map source); - - public abstract JobTypeEnum getJobType(); - public abstract void setJobType(JobTypeEnum jobType); - - public abstract String getSubmitUser(); - - public abstract Map getVariables(); - public abstract void setVariables(Map variables); - - - public abstract Map getConfiguration(); - public abstract void setConfiguration(Map configuration); - - -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/ReadJob.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/ReadJob.java deleted file mode 100644 index 6ceb08335..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/ReadJob.java +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -import java.util.List; -import java.util.Map; - -/** - * Created by peacewong on 2019/11/2. - */ -public interface ReadJob { - - Map getSharedNodesInfo(); - - String getSharedKey(String value); - - void setSharedNodesInfo(Map sharedNodesInfo); - - String[] getShareNodeIds(); -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/SharedJob.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/SharedJob.java deleted file mode 100644 index 7fb84c91a..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/SharedJob.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -/** - * Created by peacewong on 2019/11/2. - */ -public interface SharedJob { - - int getSharedNum(); - - - String getSharedKey(); -} \ No newline at end of file diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/SignalSharedJob.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/SignalSharedJob.java deleted file mode 100644 index 13a9b8b44..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/SignalSharedJob.java +++ /dev/null @@ -1,42 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.job; - -/** - * Created by enjoyyin on 2019/11/13. - */ -public interface SignalSharedJob extends SharedJob { - - String PREFIX = "signal."; - - JobSignalKeyCreator getSignalKeyCreator(); - - void setSignalKeyCreator(JobSignalKeyCreator signalKeyCreator); - - @Override - default int getSharedNum() { - return -1; - } - - @Override - default String getSharedKey() { - return PREFIX + getSignalKeyCreator().getSignalKeyBySignalSharedJob(this) + "." + getMsgSaveKey(); - } - - String getMsgSaveKey(); -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/CodeParser.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/CodeParser.java deleted file mode 100644 index f4c254e8a..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/CodeParser.java +++ /dev/null @@ -1,235 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.parser; - -import com.google.gson.reflect.TypeToken; -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; -import com.webank.wedatasphere.dss.linkis.node.execution.job.CommonLinkisJob; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils; -import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource; -import com.webank.wedatasphere.linkis.filesystem.WorkspaceClientFactory; -import com.webank.wedatasphere.linkis.filesystem.request.WorkspaceClient; -import com.webank.wedatasphere.linkis.filesystem.response.ScriptFromBMLResponse; -import org.apache.commons.lang.StringUtils; - -import java.util.*; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -/** - * Created by peacewong on 2019/11/2. - */ -public class CodeParser implements JobParser { - - private final static Pattern pb = Pattern.compile("((project)|(flow)|(node))://[^\\s\"]+[$\\s]{0,1}", Pattern.CASE_INSENSITIVE); - - @Override - public void parseJob(Job job) throws Exception{ - if (! ( job instanceof CommonLinkisJob) ) { - return ; - } - CommonLinkisJob linkisAppjointJob = (CommonLinkisJob) job; - Map script = LinkisJobExecutionUtils.gson.fromJson(linkisAppjointJob.getCode(), new TypeToken>() {}.getType()); - List jobResourceList = linkisAppjointJob.getJobResourceList(); - BMLResource scriptResource = null; - if (script == null) { - throw new LinkisJobExecutionErrorException(90102,"Script is empty"); - } - String fileName = (String) script.get("script"); - for(BMLResource bmlResource : jobResourceList){ - if(bmlResource.getFileName().equals(fileName)){ - scriptResource = bmlResource; - break; - } - } - if(null == scriptResource) { - throw new LinkisJobExecutionErrorException(90102,"Failed to get script resource"); - } - - Map executionParams = getExecutionParams(scriptResource, linkisAppjointJob); - - if (executionParams.get("executionCode") != null) { - - String executionCode = (String) executionParams.get("executionCode"); - linkisAppjointJob.getLogObj().info("************************************SUBMIT CODE************************************"); - linkisAppjointJob.getLogObj().info(executionCode); - linkisAppjointJob.getLogObj().info("************************************SUBMIT CODE************************************"); - - //get ResourceNames - ArrayList resourceNames = getResourceNames(executionCode); - - //get ResourceList - - ArrayList bmlResourceArrayList = getResourcesByNames(resourceNames, linkisAppjointJob); - - //parsedCode - linkisAppjointJob.setCode(replaceCodeResourceNames(executionCode, resourceNames, bmlResourceArrayList)); - - - //put resources - if (linkisAppjointJob.getRuntimeParams() != null) { - linkisAppjointJob.getRuntimeParams().put("resources", bmlResourceArrayList); - } - } - if (executionParams.get("params") != null && executionParams.get("params") instanceof Map) { - if (linkisAppjointJob.getParams() != null) { - linkisAppjointJob.getParams().putAll( (Map)executionParams.get("params")); - } - } - - } - - private Map getExecutionParams(BMLResource bmlResource, CommonLinkisJob linkisAppjointJob) { - Map map = new HashMap<>(); - WorkspaceClient client = WorkspaceClientFactory.getClient(linkisAppjointJob.getSubmitUser(), LinkisJobExecutionConfiguration.LINKIS_AUTHOR_USER_TOKEN.getValue(linkisAppjointJob.getJobProps()), LinkisJobExecutionConfiguration.LINKIS_URL.getValue(linkisAppjointJob.getJobProps())); - ScriptFromBMLResponse response = client.requestOpenScriptFromBML(bmlResource.getResourceId(), bmlResource.getVersion(), bmlResource.getFileName()); - map.put("executionCode", response.scriptContent()); - map.put("params", response.metadata()); - return map; - } - - private ArrayList getResourceNames(String code){ - ArrayList bmlResourceNames = new ArrayList(); - Matcher mb = pb.matcher(code); - while (mb.find()) { - bmlResourceNames.add(mb.group().trim()); - } - return bmlResourceNames; - } - - - /** - * 1.Find the project file used in the script - * 2.Find the node file used in the script - * 3.Recursively find the flow file used in the script - * 4.Replace file name with prefixed name - * @param resourceNames - * @param linkisAppjointJob - * @return - */ - private ArrayList getResourcesByNames(ArrayList resourceNames, CommonLinkisJob linkisAppjointJob) { - - ArrayList bmlResourceArrayList = new ArrayList<>(); - - String jobName = linkisAppjointJob.getJobName(); - String flowName = linkisAppjointJob.getSource().get("flowName"); - String projectName = linkisAppjointJob.getSource().get("projectName"); - - - List projectResourceList = linkisAppjointJob.getProjectResourceList(); - - - List jobResourceList = linkisAppjointJob.getJobResourceList(); - for (String resourceName : resourceNames) { - String[] resourceNameSplit = resourceName.split("://"); - String prefix = resourceNameSplit[0].toLowerCase(); - String fileName = resourceNameSplit[1]; - BMLResource resource = null; - String afterFileName = fileName; - switch (prefix) { - case "project": - resource = findResource(projectResourceList, fileName); - afterFileName = LinkisJobExecutionConfiguration.PROJECT_PREFIX + "_" + projectName + "_" + fileName; - break; - case "flow": - resource = findFlowResource(linkisAppjointJob, fileName, flowName); - break; - case "node": - resource = findResource(jobResourceList, fileName); - afterFileName = LinkisJobExecutionConfiguration.JOB_PREFIX + "_" + jobName + "_" + fileName; - break; - default: - } - if (null == resource) { - linkisAppjointJob.getLogObj().error("Failed to find the " + prefix + " resource file of " + fileName); - throw new RuntimeException("Failed to find the " + prefix + " resource file of " + fileName); - } - if (!afterFileName.equals(fileName)) { - resource.setFileName(afterFileName); - } - bmlResourceArrayList.add(resource); - } - return bmlResourceArrayList; - } - - - /** - * Recursively find the flow file used in the script - * Recursive exit condition is top-level flow - * - */ - private BMLResource findFlowResource(CommonLinkisJob linkisAppjointJob, String fileName, String flowName) { - - String fullFlowName = ""; - Map> fLowNameAndResources = linkisAppjointJob.getFlowNameAndResources(); - if (fLowNameAndResources == null){ - return null; - } - Optional>> first = fLowNameAndResources.entrySet().stream().filter(fLowNameAndResource -> fLowNameAndResource.getKey().endsWith(flowName + LinkisJobExecutionConfiguration.RESOURCES_NAME)).findFirst(); - - if(first.isPresent()){ - fullFlowName = first.get().getKey(); - BMLResource resource = findResource(first.get().getValue(), fileName); - if (resource != null) { - resource.setFileName(flowName + "_" + fileName); - return resource; - } - } - - String firstFlow = "flow." + flowName + LinkisJobExecutionConfiguration.RESOURCES_NAME; - if (firstFlow.equals(fullFlowName)) { - return null; - } - //getParentFlowName:flow.flows1.test.resources return:flows1 - String parentFlowName = StringUtils.substringAfterLast(StringUtils.substringBefore(fullFlowName, "." + flowName - + LinkisJobExecutionConfiguration.RESOURCES_NAME), "."); - if (StringUtils.isEmpty(parentFlowName)) { - return null; - } - - return findFlowResource(linkisAppjointJob, fileName, parentFlowName); - } - - - private String replaceCodeResourceNames(String code, ArrayList resourceNameList, ArrayList resourceList){ - if(resourceList.size() != resourceNameList.size()){ - throw new RuntimeException("Failed to parsed resource file"); - } - - String[] names = resourceNameList.toArray(new String[]{}); - - String[] afterNames = new String[resourceList.size()]; - for (int i=0 ; i < afterNames.length ; i++){ - afterNames[i] = resourceList.get(i).getFileName(); - } - return StringUtils.replaceEach(code, names, afterNames); - } - - private BMLResource findResource(List resourceArrayList, String fileName){ - if(resourceArrayList != null && !resourceArrayList.isEmpty()) { - for(BMLResource resource : resourceArrayList){ - if(resource.getFileName().equals(fileName)){ - return resource; - } - } - } - return null; - } -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParamsParser.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParamsParser.java deleted file mode 100644 index 27174fbcc..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParamsParser.java +++ /dev/null @@ -1,89 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.parser; - -import com.google.gson.reflect.TypeToken; -import com.webank.wedatasphere.dss.linkis.node.execution.WorkflowContext; -import com.webank.wedatasphere.dss.linkis.node.execution.job.JobSignalKeyCreator; -import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.dss.linkis.node.execution.job.SignalSharedJob; -import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import java.util.Collection; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -/** - * Created by johnnwang on 2019/11/3. - */ -public class JobParamsParser implements JobParser { - private static final Logger LOGGER = LoggerFactory.getLogger(JobParamsParser.class); - private JobSignalKeyCreator signalKeyCreator; - - public JobSignalKeyCreator getSignalKeyCreator() { - return signalKeyCreator; - } - - public void setSignalKeyCreator(JobSignalKeyCreator signalKeyCreator) { - this.signalKeyCreator = signalKeyCreator; - } - - @Override - public void parseJob(Job job) throws Exception { - - if (job instanceof LinkisJob) { - - LinkisJob linkisJob = (LinkisJob) job; - - linkisJob.getLogObj().info("Start to put variable and configuration"); - //put variable - Map flowVariables = linkisJob.getVariables(); - putParamsMap(job.getParams(), "variable", flowVariables); - //put signal info - Map sharedValue = WorkflowContext.getAppJointContext() - .getSubMapByPrefix(SignalSharedJob.PREFIX + this.getSignalKeyCreator().getSignalKeyByJob(job)); - if (sharedValue != null) { - Collection values = sharedValue.values(); - for(Object value : values){ - List> list = LinkisJobExecutionUtils.gson.fromJson(value.toString(), List.class); - Map totalMap = new HashMap<>(); - for (Map kv : list) { - totalMap.putAll(kv); - } - putParamsMap(job.getParams(), "variable", totalMap); - } - } - // put configuration - Map configuration = linkisJob.getConfiguration(); - putParamsMap(job.getParams(), "configuration", configuration); - linkisJob.getLogObj().info("Finished to put variable and configuration"); - } - } - - private void putParamsMap(Map params, String key, Map value) { - if (null == value) return; - if (params.get(key) != null) { - ((Map) params.get(key)).putAll(value); - } else { - params.put(key, value); - } - } -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobRuntimeParamsParser.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobRuntimeParamsParser.java deleted file mode 100644 index 6c071f1ee..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobRuntimeParamsParser.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.parser; - -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.dss.linkis.node.execution.job.ReadJob; - -/** - * Created by peacewong on 2019/11/3. - */ -public class JobRuntimeParamsParser implements JobParser{ - - @Override - public void parseJob(Job job) throws Exception{ - if(job instanceof LinkisJob) { - job.getRuntimeParams().put(LinkisJobExecutionConfiguration.LINKIS_SUBMIT_USER,((LinkisJob) job).getSubmitUser()); - } - if (job instanceof ReadJob) { - //put the shared nods be read node need - job.getRuntimeParams() - .put(LinkisJobExecutionConfiguration.WORKFLOW_SHARED_NODES_JOBIDS, ((ReadJob) job).getSharedNodesInfo()); - } - } -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/BuildJobAction.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/BuildJobAction.java deleted file mode 100644 index 1168b52ca..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/BuildJobAction.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.service; - - -import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; - -/** - * Created by peacewong on 2019/9/21. - */ -public interface BuildJobAction { - - public JobExecuteAction getJobAction(Job job) throws LinkisJobExecutionErrorException; - -} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/BuildJobActionImpl.java b/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/BuildJobActionImpl.java deleted file mode 100644 index 8927be8fc..000000000 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/BuildJobActionImpl.java +++ /dev/null @@ -1,65 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.node.execution.service.impl; - -import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; -import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob; -import com.webank.wedatasphere.dss.linkis.node.execution.service.BuildJobAction; -import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; -import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; - -import java.util.HashMap; -import java.util.Map; - - -/** - * Created by peacewong on 2019/11/3. - */ -public class BuildJobActionImpl implements BuildJobAction { - - private static BuildJobAction buildJobAction = new BuildJobActionImpl(); - - private BuildJobActionImpl(){ - - } - - public static BuildJobAction getbuildJobAction(){ - return buildJobAction; - } - - @Override - public JobExecuteAction getJobAction(Job job) throws LinkisJobExecutionErrorException { - - JobExecuteAction.Builder builder = JobExecuteAction.builder().setCreator(LinkisJobExecutionConfiguration.LINKIS_JOB_CREATOR.getValue(job.getJobProps())) - .addExecuteCode(job.getCode()) - .setEngineTypeStr(job.getEngineType()) - .setRunTypeStr(job.getRunType()) - .setUser(job.getUser()) - .setParams(job.getParams()) - .setRuntimeParams(job.getRuntimeParams()); - if(job instanceof LinkisJob){ - Map source = new HashMap<>(); - source.putAll(((LinkisJob) job).getSource()); - builder = builder.setSource(source); - } - return builder.build(); - } - - -} diff --git a/dss-orchestrator/dss-orchestrator-common/pom.xml b/dss-orchestrator/dss-orchestrator-common/pom.xml new file mode 100644 index 000000000..c2778c1cb --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/pom.xml @@ -0,0 +1,48 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator-common + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + + + \ No newline at end of file diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestration.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestration.java new file mode 100644 index 000000000..7fa625b75 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestration.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.entity; + + +public interface DSSOrchestration { + + Long getId(); + + String getName(); + + String getDescription(); + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestratorInfo.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestratorInfo.java new file mode 100644 index 000000000..48584bef1 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestratorInfo.java @@ -0,0 +1,208 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.entity; + +import java.util.Date; +import java.util.List; + + +public class DSSOrchestratorInfo implements DSSOrchestration { + + private Long id; + + private String name; + + private String type; + + private String desc; + + private String creator; + + private Date createTime; + + private String uses; + + private String appConnName; + + private Long projectId; + + private String uuid; + + private String secondaryType; + + public DSSOrchestratorInfo() { + + } + + public DSSOrchestratorInfo(String name, String type, + String desc, String creator, + Date createTime, String uses, + String appConnName, Long projectId, String secondaryType, + List linkedAppConnNames, String comment) { + this.name = name; + this.type = type; + this.desc = desc; + this.creator = creator; + this.createTime = createTime; + this.uses = uses; + this.appConnName = appConnName; + this.projectId = projectId; + this.secondaryType = secondaryType; + this.linkedAppConnNames = linkedAppConnNames; + this.comment = comment; + } + + public String getSecondaryType() { + return secondaryType; + } + + public void setSecondaryType(String secondaryType) { + this.secondaryType = secondaryType; + } + + + public String getUUID() { + return uuid; + } + + public void setUUID(String uuid) { + this.uuid = uuid; + } + + + /** + * 用来存储该编排可以支持的节点类型名称 + */ + private List linkedAppConnNames; + + public List getLinkedAppConnNames() { + return linkedAppConnNames; + } + + public void setLinkedAppConnNames(List linkedAppConnNames) { + this.linkedAppConnNames = linkedAppConnNames; + } + + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + private String comment; + + public String getAppConnName() { + return appConnName; + } + + public void setAppConnName(String appConnName) { + this.appConnName = appConnName; + } + + @Override + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + @Override + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public String getDesc() { + return desc; + } + + @Override + public String getDescription() { + return desc; + } + + public void setDesc(String desc) { + this.desc = desc; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public long getProjectId() { + return projectId; + } + + public void setProjectId(long projectId) { + this.projectId = projectId; + } + + @Override + public String toString() { + return "DSSOrchestratorInfo{" + + "id=" + id + + ", name='" + name + '\'' + + ", type='" + type + '\'' + + ", desc='" + desc + '\'' + + ", creator='" + creator + '\'' + + ", createTime=" + createTime + + ", uses='" + uses + '\'' + + ", appConnName='" + appConnName + '\'' + + ", projectId=" + projectId + + ", uuid='" + uuid + '\'' + + ", secondaryType='" + secondaryType + '\'' + + ", linkedAppConnNames=" + linkedAppConnNames + + ", comment='" + comment + '\'' + + '}'; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestratorVersion.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestratorVersion.java new file mode 100644 index 000000000..44ed8f375 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/DSSOrchestratorVersion.java @@ -0,0 +1,167 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.entity; + +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import java.util.Date; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class DSSOrchestratorVersion { + private final static Logger LOGGER = LoggerFactory.getLogger(DSSOrchestratorVersion.class); + + + private Long id; + private Long orchestratorId; + private Long appId; + private Long projectId; + private String source; + private String version; + private String comment; + private Date updateTime; + private String updater; + private String content; + private String contextId; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public Long getAppId() { + return appId; + } + + public void setAppId(Long appId) { + this.appId = appId; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public String getUpdater() { + return updater; + } + + public void setUpdater(String updater) { + this.updater = updater; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getContent() { + return content; + } + + public void setContent(String content) { + this.content = content; + } + + public String getContextId() { + return contextId; + } + + public void setContextId(String contextId) { + this.contextId = contextId; + } + + /** + * 提取出contextId + */ + public void setFormatContextId(String contextId) { + try { + if (contextId == null || "".equals(contextId.trim()) || !contextId.contains("value") || !contextId.contains("contextId")) { + this.contextId = contextId; + } else { + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(contextId).getAsJsonObject(); + String tempValue = jsonObject.get("value").getAsString(); + String tempContextId = parser.parse(tempValue).getAsJsonObject().get("contextId").getAsString(); + this.contextId = tempContextId; + } + } catch (Exception e) { + LOGGER.error("setFormatContextIdError-", e); + this.contextId = contextId; + } + } + + @Override + public String toString() { + return "DSSOrchestratorVersion{" + + "id=" + id + + ", orchestratorId=" + orchestratorId + + ", appId=" + appId + + ", projectId=" + projectId + + ", source='" + source + '\'' + + ", version='" + version + '\'' + + ", comment='" + comment + '\'' + + ", updateTime=" + updateTime + + ", updater='" + updater + '\'' + + ", content='" + content + '\'' + + ", contextId='" + contextId + '\'' + + '}'; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorInfo.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorInfo.java new file mode 100644 index 000000000..2382c17d6 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorInfo.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.entity; + + +public class OrchestratorInfo { + + private Long orchestratorId; + + private Long orchestratorVersionId; + + public OrchestratorInfo(Long orchestratorId, Long orchestratorVersionId) { + this.orchestratorId = orchestratorId; + this.orchestratorVersionId = orchestratorVersionId; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public Long getOrchestratorVersionId() { + return orchestratorVersionId; + } + + public void setOrchestratorVersionId(Long orchestratorVersionId) { + this.orchestratorVersionId = orchestratorVersionId; + } + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorUser.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorUser.java new file mode 100644 index 000000000..925016bab --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorUser.java @@ -0,0 +1,95 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.entity; + +import java.util.Date; + + +public class OrchestratorUser { + + private Long id; + + private Long workspaceId; + + private Long projectId; + + private Long orchestratorId; + + private String username; + + //权限等级：0:私密；1：公开 + private Integer priv; + + private Date lastUpdateTime; + + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getWorkspaceId() { + return workspaceId; + } + + public void setWorkspaceId(Long workspaceId) { + this.workspaceId = workspaceId; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public Integer getPriv() { + return priv; + } + + public void setPriv(Integer priv) { + this.priv = priv; + } + + public Date getLastUpdateTime() { + return lastUpdateTime; + } + + public void setLastUpdateTime(Date lastUpdateTime) { + this.lastUpdateTime = lastUpdateTime; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorVo.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorVo.java new file mode 100644 index 000000000..fae4b94a4 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/entity/OrchestratorVo.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.entity; + + + + +public class OrchestratorVo { + + DSSOrchestratorInfo dssOrchestratorInfo; + + public DSSOrchestratorVersion getDssOrchestratorVersion() { + return dssOrchestratorVersion; + } + + public void setDssOrchestratorVersion(DSSOrchestratorVersion dssOrchestratorVersion) { + this.dssOrchestratorVersion = dssOrchestratorVersion; + } + + DSSOrchestratorVersion dssOrchestratorVersion; + + public DSSOrchestratorInfo getDssOrchestratorInfo() { + return dssOrchestratorInfo; + } + + public void setDssOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo = dssOrchestratorInfo; + } + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestConvertOrchestrations.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestConvertOrchestrations.java new file mode 100644 index 000000000..0496ca767 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestConvertOrchestrations.java @@ -0,0 +1,72 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.entity.DSSWorkspace; +import com.webank.wedatasphere.dss.common.entity.project.Project; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import java.util.List; + + +public class RequestConvertOrchestrations { + + private String userName; + private List orcAppIds; + private Project project; + private DSSWorkspace workspace; + private List dssLabels; + + public void setOrcAppIds(List orcAppIds) { + this.orcAppIds = orcAppIds; + } + + public List getOrcAppIds() { + return orcAppIds; + } + + public Project getProject() { + return project; + } + + public void setProject(Project project) { + this.project = project; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public DSSWorkspace getWorkspace() { + return workspace; + } + + public void setWorkspace(DSSWorkspace workspace) { + this.workspace = workspace; + } + + public List getDSSLabels() { + return dssLabels; + } + + public void setDSSLabels(List dssLabels) { + this.dssLabels = dssLabels; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestCreateOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestCreateOrchestrator.java new file mode 100644 index 000000000..ca37f549f --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestCreateOrchestrator.java @@ -0,0 +1,112 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import java.util.List; + + +public class RequestCreateOrchestrator { + + private String userName; + private String workspaceName; + private String projectName; + private Long projectId; + private String description; + private DSSOrchestratorInfo dssOrchestratorInfo; + private List dssLabels; + + public RequestCreateOrchestrator(String userName, + String workspaceName, + String projectName, + Long projectId, + String description, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) { + this.userName = userName; + this.workspaceName = workspaceName; + this.projectName = projectName; + this.projectId = projectId; + this.description = description; + this.dssOrchestratorInfo = dssOrchestratorInfo; + this.dssLabels = dssLabels; + } + + + + + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public DSSOrchestratorInfo getDssOrchestratorInfo() { + return dssOrchestratorInfo; + } + + public void setDssOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo = dssOrchestratorInfo; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestDeleteOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestDeleteOrchestrator.java new file mode 100644 index 000000000..86c4c497c --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestDeleteOrchestrator.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import java.util.List; + + +public class RequestDeleteOrchestrator { + private String userName; + private String workspaceName; + private String projectName; + private Long orchestratorId; + private List dssLabels; + + + public RequestDeleteOrchestrator(String userName, + String workspaceName, + String projectName, + Long orchestratorId, + List dssLabels) { + this.userName = userName; + this.workspaceName = workspaceName; + this.projectName = projectName; + this.orchestratorId = orchestratorId; + this.dssLabels = dssLabels; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestExportOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestExportOrchestrator.java new file mode 100644 index 000000000..33039d673 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestExportOrchestrator.java @@ -0,0 +1,115 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import java.util.List; + + +public class RequestExportOrchestrator { + + private String userName; + private String workspaceName; + private Long orchestratorId; + private Long orcVersionId; + private String projectName; + private List dssLabels; + private Boolean addOrcVersion; + private String workspaceStr; + + public RequestExportOrchestrator(String userName, + String workspaceName, + Long orchestratorId, + Long orcVersionId, + String projectName, + List dssLabels, + Boolean addOrcVersion, + String workspaceStr) { + this.userName = userName; + this.workspaceName = workspaceName; + this.orcVersionId = orcVersionId; + this.orchestratorId = orchestratorId; + this.projectName = projectName; + this.dssLabels = dssLabels; + this.addOrcVersion = addOrcVersion; + this.workspaceStr = workspaceStr; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public Long getOrcVersionId() { + return orcVersionId; + } + + public void setOrcVersionId(Long orcVersionId) { + this.orcVersionId = orcVersionId; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + public Boolean getAddOrcVersion() { + return addOrcVersion; + } + + public void setAddOrcVersion(Boolean addOrcVersion) { + this.addOrcVersion = addOrcVersion; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public String getWorkspaceStr() { + return workspaceStr; + } + + public void setWorkspaceStr(String workspaceStr) { + this.workspaceStr = workspaceStr; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestFrameworkConvertOrchestration.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestFrameworkConvertOrchestration.java new file mode 100644 index 000000000..310c283c9 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestFrameworkConvertOrchestration.java @@ -0,0 +1,91 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.entity.DSSWorkspace; +import java.util.List; +import java.util.Map; + + +public class RequestFrameworkConvertOrchestration { + + private String userName; + private DSSWorkspace workspace; + private Long orcAppId; + private List orcIds; + private Map labels; + private boolean convertAllOrcs; + private String comment; + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public DSSWorkspace getWorkspace() { + return workspace; + } + + public void setWorkspace(DSSWorkspace workspace) { + this.workspace = workspace; + } + + public Long getOrcAppId() { + return orcAppId; + } + + public void setOrcAppId(Long orcAppId) { + this.orcAppId = orcAppId; + } + + public List getOrcIds() { + return orcIds; + } + + public void setOrcIds(List orcIds) { + this.orcIds = orcIds; + } + + public Map getLabels() { + return labels; + } + + public void setLabels(Map labels) { + this.labels = labels; + } + + public boolean isConvertAllOrcs() { + return convertAllOrcs; + } + + public void setConvertAllOrcs(boolean convertAllOrcs) { + this.convertAllOrcs = convertAllOrcs; + } + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestFrameworkConvertOrchestrationStatus.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestFrameworkConvertOrchestrationStatus.java new file mode 100644 index 000000000..8a85bba5f --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestFrameworkConvertOrchestrationStatus.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + + +public class RequestFrameworkConvertOrchestrationStatus { + + private String id; + + public RequestFrameworkConvertOrchestrationStatus() { + } + + public RequestFrameworkConvertOrchestrationStatus(String id) { + this.id = id; + } + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestImportOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestImportOrchestrator.java new file mode 100644 index 000000000..01f04e64a --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestImportOrchestrator.java @@ -0,0 +1,129 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; + + +public class RequestImportOrchestrator { + + + private String userName; + private String workspaceName; + private String projectName; + private Long projectId; + private String resourceId; + private String bmlVersion; + private String orchestratorName; + private List dssLabels; + private String workspaceStr; + + public RequestImportOrchestrator(String userName, + String workspaceName, + String projectName, + Long projectId, + String resourceId, + String bmlVersion, + String orchestratorName, + List dssLabels, + String workspaceStr) { + this.userName = userName; + this.workspaceName = workspaceName; + this.projectName = projectName; + this.projectId = projectId; + this.resourceId = resourceId; + this.bmlVersion = bmlVersion; + this.dssLabels = dssLabels; + this.orchestratorName = orchestratorName; + this.workspaceStr = workspaceStr; + } + + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getResourceId() { + return resourceId; + } + + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + public String getBmlVersion() { + return bmlVersion; + } + + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + public String getOrchestratorName() { + return orchestratorName; + } + + public void setOrchestratorName(String orchestratorName) { + this.orchestratorName = orchestratorName; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + public String getWorkspaceStr() { + return workspaceStr; + } + + public void setWorkspaceStr(String workspaceStr) { + this.workspaceStr = workspaceStr; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestOrchestratorVersion.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestOrchestratorVersion.java new file mode 100644 index 000000000..d71a4d7f8 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestOrchestratorVersion.java @@ -0,0 +1,69 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + + +public class RequestOrchestratorVersion { + + private String username; + private Long projectId; + private Long orchestratorId; + private String dssLabel; + + + public static RequestOrchestratorVersion newInstance(String username, Long projectId, Long orchestratorId, String dssLabel) { + RequestOrchestratorVersion requestOrchestratorVersion = new RequestOrchestratorVersion(); + requestOrchestratorVersion.setOrchestratorId(orchestratorId); + requestOrchestratorVersion.setDssLabel(dssLabel); + requestOrchestratorVersion.setProjectId(projectId); + requestOrchestratorVersion.setUsername(username); + return requestOrchestratorVersion; + } + + + public String getUsername() { + return username; + } + + public void setUsername(String username) { + this.username = username; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public String getDssLabel() { + return dssLabel; + } + + public void setDssLabel(String dssLabel) { + this.dssLabel = dssLabel; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestProjectImportOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestProjectImportOrchestrator.java new file mode 100644 index 000000000..db4d1be92 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestProjectImportOrchestrator.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; + +public class RequestProjectImportOrchestrator extends DSSOrchestratorInfo { + + private Long versionId; + + public Long getVersionId() { + return versionId; + } + + public void setVersionId(Long versionId) { + this.versionId = versionId; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestQueryOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestQueryOrchestrator.java new file mode 100644 index 000000000..04290298b --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestQueryOrchestrator.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import java.util.List; + + +public class RequestQueryOrchestrator { + private List orchestratorIds; + + public RequestQueryOrchestrator(List orchestratorIds) { + this.orchestratorIds = orchestratorIds; + } + + public List getOrchestratorIds() { + return orchestratorIds; + } + + public void setOrchestratorIds(List orchestratorIds) { + this.orchestratorIds = orchestratorIds; + } + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestUpdateOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestUpdateOrchestrator.java new file mode 100644 index 000000000..bade06020 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/RequestUpdateOrchestrator.java @@ -0,0 +1,76 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; + +import java.util.List; + + +public class RequestUpdateOrchestrator { + private String userName; + private String workspaceName; + private DSSOrchestratorInfo dssOrchestratorInfo; + private List dssLabels; + + + public RequestUpdateOrchestrator(String userName, + String workspaceName, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) { + this.userName = userName; + this.workspaceName = workspaceName; + this.dssOrchestratorInfo = dssOrchestratorInfo; + this.dssLabels = dssLabels; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public DSSOrchestratorInfo getDssOrchestratorInfo() { + return dssOrchestratorInfo; + } + + public void setDssOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo = dssOrchestratorInfo; + } + + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseConvertOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseConvertOrchestrator.java new file mode 100644 index 000000000..111f1ddd1 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseConvertOrchestrator.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + + +public class ResponseConvertOrchestrator { + + private String id; + private ResponseOperateOrchestrator response; + + public ResponseConvertOrchestrator() { + } + + public ResponseConvertOrchestrator(String id, ResponseOperateOrchestrator response) { + this.id = id; + this.response = response; + } + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public ResponseOperateOrchestrator getResponse() { + return response; + } + + public void setResponse(ResponseOperateOrchestrator response) { + this.response = response; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseOperateOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseOperateOrchestrator.java new file mode 100644 index 000000000..d862e2b86 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseOperateOrchestrator.java @@ -0,0 +1,96 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; + + +public class ResponseOperateOrchestrator { + + private JobStatus jobStatus; + private String message; + + public JobStatus getJobStatus() { + return jobStatus; + } + + public void setJobStatus(JobStatus jobStatus) { + this.jobStatus = jobStatus; + } + + public String getMessage() { + return message; + } + + public void setMessage(String message) { + this.message = message; + } + + public boolean isCompleted() { + return isSucceed() || isFailed(); + } + + public boolean isSucceed() { + return jobStatus == JobStatus.Success; + } + + public boolean isFailed() { + return jobStatus == JobStatus.Failed; + } + + public boolean isRunning() { + return jobStatus == JobStatus.Running; + } + + public static ResponseOperateOrchestrator success() { + ResponseOperateOrchestrator response = new ResponseOperateOrchestrator(); + response.setJobStatus(JobStatus.Success); + return response; + } + + public static ResponseOperateOrchestrator success(String message) { + ResponseOperateOrchestrator response = new ResponseOperateOrchestrator(); + response.setJobStatus(JobStatus.Success); + response.setMessage(message); + return response; + } + + public static ResponseOperateOrchestrator failed() { + ResponseOperateOrchestrator response = new ResponseOperateOrchestrator(); + response.setJobStatus(JobStatus.Failed); + return response; + } + + public static ResponseOperateOrchestrator failed(String message) { + ResponseOperateOrchestrator response = new ResponseOperateOrchestrator(); + response.setJobStatus(JobStatus.Failed); + response.setMessage(message); + return response; + } + + public static ResponseOperateOrchestrator running() { + ResponseOperateOrchestrator response = new ResponseOperateOrchestrator(); + response.setJobStatus(JobStatus.Running); + return response; + } + + public static ResponseOperateOrchestrator inited() { + ResponseOperateOrchestrator response = new ResponseOperateOrchestrator(); + response.setJobStatus(JobStatus.Inited); + return response; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseOrchetratorVersion.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseOrchetratorVersion.java new file mode 100644 index 000000000..99f0f1478 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseOrchetratorVersion.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; + +import java.util.List; + + +public class ResponseOrchetratorVersion { + + private Long projectId; + private Long orchestratorId; + private List orchestratorVersions; + + public ResponseOrchetratorVersion(Long projectId, Long orchestratorId, List orchestratorVersions) { + this.projectId = projectId; + this.orchestratorId = orchestratorId; + this.orchestratorVersions = orchestratorVersions; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public Long getOrchestratorId() { + return orchestratorId; + } + + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + public List getOrchestratorVersions() { + return orchestratorVersions; + } + + public void setOrchestratorVersions(List orchestratorVersions) { + this.orchestratorVersions = orchestratorVersions; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseQueryOrchestrator.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseQueryOrchestrator.java new file mode 100644 index 000000000..31e70372f --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/protocol/ResponseQueryOrchestrator.java @@ -0,0 +1,42 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.protocol; + + +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; + +import java.util.List; + + +public class ResponseQueryOrchestrator { + + private List orchestratorVoes; + + public ResponseQueryOrchestrator(List orchestratorVoes) { + this.orchestratorVoes = orchestratorVoes; + } + + public List getOrchestratorVoes() { + return orchestratorVoes; + } + + public void setOrchestratorVoes(List orchestratorVoes) { + this.orchestratorVoes = orchestratorVoes; + } + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/DefaultOrchestratorCreateRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/DefaultOrchestratorCreateRequestRef.java new file mode 100644 index 000000000..b3137b560 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/DefaultOrchestratorCreateRequestRef.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class DefaultOrchestratorCreateRequestRef extends CommonRequestRefImpl implements OrchestratorCreateRequestRef { + + + private DSSOrchestratorInfo dssOrchestratorInfo; + + @Override + public DSSOrchestratorInfo getDSSOrchestratorInfo() { + return dssOrchestratorInfo; + } + + @Override + public void setDssOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo = dssOrchestratorInfo; + } + + + @Override + public String toString() { + return "WorkflowOrchestratorCreateRequestRef{" + + "username='" + userName + '\'' + + ", workspaceName='" + workspaceName + '\'' + + ", projectName='" + projectName + '\'' + + ", projectId=" + projectId + + ", contextIDStr='" + contextID + '\'' + + ", dssLabels=" + dssLabelList + + ", name='" + name + '\'' + + ", type='" + type + '\'' + + '}'; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCopyRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCopyRequestRef.java new file mode 100644 index 000000000..04c64ae77 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCopyRequestRef.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.CopyRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; + + +public interface OrchestratorCopyRequestRef extends CopyRequestRef, CommonRequestRef { + + void setCopyOrcAppId(long appId); + + void setCopyOrcVersionId(long orcId); + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCopyResponseRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCopyResponseRef.java new file mode 100644 index 000000000..995377861 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCopyResponseRef.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public interface OrchestratorCopyResponseRef extends ResponseRef { + + long getCopyOrcId(); + + long getCopyTargetAppId(); + + String getCopyTargetContent(); + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCreateRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCreateRequestRef.java new file mode 100644 index 000000000..f8992b293 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCreateRequestRef.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.standard.app.development.ref.CreateRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; + + + +public interface OrchestratorCreateRequestRef extends CreateRequestRef, CommonRequestRef { + + void setDssOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo); + + DSSOrchestratorInfo getDSSOrchestratorInfo(); +} + diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCreateResponseRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCreateResponseRef.java new file mode 100644 index 000000000..4e05ee183 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorCreateResponseRef.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public interface OrchestratorCreateResponseRef extends ResponseRef { + + /** + * 返回第三方应用和orchestrator关联的Id + * @return + */ + + + void setOrcId(Long orcId); + + Long getOrcId(); + + + /** + * 返回第三方应用创建时返回的Content的内容 + * @return + */ + String getContent(); + + void setContent(String content); + + + + + + Long getOrchestratorVersionId(); + + void setOrchestratorVersionId(Long versionId); +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorDeleteRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorDeleteRequestRef.java new file mode 100644 index 000000000..fdc369dcb --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorDeleteRequestRef.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.DeleteRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; + + +public interface OrchestratorDeleteRequestRef extends DeleteRequestRef, CommonRequestRef { + + /** + * 检索数据库中关联的第三方应用的id + * @param appId + */ + void setAppId(Long appId); + + Long getAppId(); + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorExportRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorExportRequestRef.java new file mode 100644 index 000000000..e2817cf84 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorExportRequestRef.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; + + +public interface OrchestratorExportRequestRef extends ExportRequestRef, CommonRequestRef { + + void setAppId(Long appId); + + Long getAppId(); + + void setOrchestratorVersionId(Long orchestratorVersionId); + + Long getOrchestratorVersionId(); + + boolean getAddOrcVersionFlag(); + + void setAddOrcVersionFlag(boolean addOrcVersion); + + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorExportResponseRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorExportResponseRef.java new file mode 100644 index 000000000..3e3059d58 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorExportResponseRef.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public interface OrchestratorExportResponseRef extends ResponseRef { + String getResourceId(); + + void setResourceId(String resourceId); + + String getBmlVersion(); + + void setBmlVersion(String bmlVersion); + + Long getOrchestratorVersionId(); + + void setOrchestratorVersionId(Long orchestratorVersionId); +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorFrameworkAppConn.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorFrameworkAppConn.java new file mode 100644 index 000000000..424a5e720 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorFrameworkAppConn.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; + + +public interface OrchestratorFrameworkAppConn extends AppConn { + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorImportRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorImportRequestRef.java new file mode 100644 index 000000000..749376818 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorImportRequestRef.java @@ -0,0 +1,42 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.standard.app.development.ref.ImportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; + + +public interface OrchestratorImportRequestRef extends ImportRequestRef, CommonRequestRef { + + void setResourceId(String resourceId); + + String getResourceId(); + + void setBmlVersion(String bmlVersion); + + String getBmlVersion(); + + void setSourceEnv(IOEnv sourceEnv); + + IOEnv getSourceEnv(); + + void setOrcVersion(String orcVersion); + + String getOrcVersion(); + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorOpenRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorOpenRequestRef.java new file mode 100644 index 000000000..a6719a986 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorOpenRequestRef.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.app.development.ref.OpenRequestRef; + +import java.util.List; + + +public interface OrchestratorOpenRequestRef extends OpenRequestRef { + void setOrchestratorId(Long orchestratorId); + Long getOrchestratorId(); + void setUserName(String userName); + String getUserName(); + void setRefAppId(Long appId); + Long getRefAppId(); + String getSecondaryType(); + void setSecondaryType(String secondaryType); + @Override + List getDSSLabels(); + void setDSSLabels(List dssLabels); +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorQueryRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorQueryRequestRef.java new file mode 100644 index 000000000..aec3e231d --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorQueryRequestRef.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; + +import java.util.List; + + +public interface OrchestratorQueryRequestRef extends CommonRequestRef { + + List getOrchestratorIdList(); + + void setOrchestratorIdList(List orchestratorIdList); + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorQueryResponseRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorQueryResponseRef.java new file mode 100644 index 000000000..aa2383690 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorQueryResponseRef.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorVo; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +import java.util.List; + + +public interface OrchestratorQueryResponseRef extends ResponseRef { + List getOrchestratorVos(); + void setOrchestratorVoList(List orchestratorVoList); +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorUpdateRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorUpdateRef.java new file mode 100644 index 000000000..44081181f --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/OrchestratorUpdateRef.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; + +import java.util.List; + + +public interface OrchestratorUpdateRef extends RequestRef { + + void setUserName(String userName); + + String getUserName(); + + void setOrcId(Long orcID); + + Long getOrcId(); + + void setDescription(String description); + + String getDescription(); + + void setUses(String uses); + + String getUses(); + + void setOrcName(String name); + + String getOrcName(); + + String getWorkspaceName(); + + void setWorkspaceName(String workspaceName); + + String getProjectName(); + + void setProjectName(String projectName); + + DSSOrchestratorInfo getOrchestratorInfo(); + + void setOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo); + + @Override + List getDSSLabels(); + + void setDSSLabels(List dssLabels); + + + + +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/OrchestratorOpenRequestRefImpl.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/OrchestratorOpenRequestRefImpl.java new file mode 100644 index 000000000..86e434655 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/OrchestratorOpenRequestRefImpl.java @@ -0,0 +1,106 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref.impl; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorOpenRequestRef; + +import java.util.List; +import java.util.Map; + + +public class OrchestratorOpenRequestRefImpl implements OrchestratorOpenRequestRef { + + private Long orchestratorId; + private String userName; + private Long appId; + private String secondaryType; + private List dssLabels; + @Override + public void setOrchestratorId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + @Override + public Long getOrchestratorId() { + return this.orchestratorId; + } + + @Override + public void setUserName(String userName) { + this.userName = userName; + } + + @Override + public String getUserName() { + return this.userName; + } + + @Override + public void setRefAppId(Long appId) { + this.appId = appId; + } + + @Override + public Long getRefAppId() { + return this.appId; + } + + @Override + public String getSecondaryType() { + return this.secondaryType; + } + + @Override + public void setSecondaryType(String secondaryType) { + this.secondaryType = secondaryType; + } + + @Override + public List getDSSLabels() { + return dssLabels; + } + + @Override + public void setDSSLabels(List dssLabels) { + this.dssLabels = dssLabels; + } + + @Override + public Object getParameter(String key) { + return null; + } + + @Override + public void setParameter(String key, Object value) { + + } + + @Override + public Map getParameters() { + return null; + } + + @Override + public String getName() { + return null; + } + + @Override + public String getType() { + return null; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/WorkflowOrchestratoDeleteRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/WorkflowOrchestratoDeleteRequestRef.java new file mode 100644 index 000000000..922a7119e --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/WorkflowOrchestratoDeleteRequestRef.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref.impl; + + +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorDeleteRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + +public class WorkflowOrchestratoDeleteRequestRef extends CommonRequestRefImpl implements OrchestratorDeleteRequestRef { + + private Long orchestratorId; + + @Override + public Long getOrcId() { + return this.orchestratorId; + } + + @Override + public void setAppId(Long appId) { + + } + + @Override + public Long getAppId() { + return null; + } + + @Override + public void setOrcId(Long orchestratorId) { + this.orchestratorId = orchestratorId; + } + + @Override + public String toString() { + return "WorkflowOrchestratoDeleteRequestRef{" + + "userName='" + userName + '\'' + + ", workspaceName='" + workspaceName + '\'' + + ", projectName='" + projectName + '\'' + + ", orchestratorId=" + orchestratorId + + ", dssLabels=" + dssLabelList + + '}'; + } +} diff --git a/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/WorkflowOrchestratorUpdateRequestRef.java b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/WorkflowOrchestratorUpdateRequestRef.java new file mode 100644 index 000000000..8957b52a2 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-common/src/main/java/com/webank/wedatasphere/dss/orchestrator/common/ref/impl/WorkflowOrchestratorUpdateRequestRef.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.common.ref.impl; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.ref.OrchestratorUpdateRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.impl.CommonRequestRefImpl; + + +public class WorkflowOrchestratorUpdateRequestRef extends CommonRequestRefImpl implements OrchestratorUpdateRef { + + private DSSOrchestratorInfo dssOrchestratorInfo; + + @Override + public void setDescription(String description) { + + } + + @Override + public String getDescription() { + return null; + } + + @Override + public void setUses(String uses) { + + } + + @Override + public String getUses() { + return null; + } + + @Override + public DSSOrchestratorInfo getOrchestratorInfo() { + return this.dssOrchestratorInfo; + } + + @Override + public void setOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo=dssOrchestratorInfo; + } + + public DSSOrchestratorInfo getDssOrchestratorInfo() { + return dssOrchestratorInfo; + } + + public void setDssOrchestratorInfo(DSSOrchestratorInfo dssOrchestratorInfo) { + this.dssOrchestratorInfo = dssOrchestratorInfo; + } + + @Override + public String toString() { + return "WorkflowOrchestratorUpdateRequestRef{" + + "userName='" + userName + '\'' + + ", workspaceName='" + workspaceName + '\'' + + ", dssOrchestratorInfo=" + dssOrchestratorInfo + + ", dssLabels=" + dssLabelList + + '}'; + } +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/pom.xml b/dss-orchestrator/dss-orchestrator-conversion-standard/pom.xml new file mode 100644 index 000000000..0b86e128c --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/pom.xml @@ -0,0 +1,72 @@ + + + + + + dss-orchestrator + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator-conversion-standard + jar + + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-orchestrator-common + ${dss.version} + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + + 8 + 8 + + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/AbstractConversionIntegrationStandard.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/AbstractConversionIntegrationStandard.java new file mode 100644 index 000000000..e8d0018c3 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/AbstractConversionIntegrationStandard.java @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.ConversionService; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.DSSToRelConversionService; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.RelToOrchestratorConversionService; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.core.AbstractAppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import java.io.IOException; + + +public abstract class AbstractConversionIntegrationStandard extends AbstractAppIntegrationStandard + implements ConversionIntegrationStandard { + + private String appConnName; + + protected abstract DSSToRelConversionService createDSSToRelConversionService(); + + protected abstract RelToOrchestratorConversionService createRelToDSSConversionService(); + + @Override + public DSSToRelConversionService getDSSToRelConversionService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createDSSToRelConversionService, DSSToRelConversionService.class); + } + + @Override + public RelToOrchestratorConversionService getRelToDSSConversionService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRelToDSSConversionService, RelToOrchestratorConversionService.class); + } + + @Override + protected void initService(T service) { + super.initService(service); + service.setAppStandard(this); + } + + @Override + public void init() throws AppStandardErrorException { + + } + + @Override + public String getAppConnName() { + return appConnName; + } + + @Override + public void setAppConnName(String appConnName) { + this.appConnName = appConnName; + } + + @Override + public void close() throws IOException { + + } +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ConversionIntegrationStandard.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ConversionIntegrationStandard.java new file mode 100644 index 000000000..63dc71bcb --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ConversionIntegrationStandard.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.DSSToRelConversionService; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.RelToOrchestratorConversionService; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.core.AppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; + + +public interface ConversionIntegrationStandard extends AppIntegrationStandard { + + DSSToRelConversionService getDSSToRelConversionService(AppInstance appInstance); + + RelToOrchestratorConversionService getRelToDSSConversionService(AppInstance appInstance); + + String getAppConnName(); + + void setAppConnName(String appConnName); + + @Override + default String getStandardName() { + return "conversionIntegrationStandard"; + } + + @Override + default int getGrade() { + return 4; + } + + @Override + default boolean isNecessary() { + return false; + } +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/operation/ConversionOperation.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/operation/ConversionOperation.java new file mode 100644 index 000000000..a52a345db --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/operation/ConversionOperation.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.operation; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.ConversionRequestRef; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.ConversionService; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.service.Operation; + + +public interface ConversionOperation extends Operation { + + void setConversionService(ConversionService conversionService); + + V convert(K ref); + + void init(); + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/operation/DSSToRelConversionOperation.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/operation/DSSToRelConversionOperation.java new file mode 100644 index 000000000..c28bf5e88 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/operation/DSSToRelConversionOperation.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.operation; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.DSSToRelConversionRequestRef; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.ConversionService; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.DSSToRelConversionService; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public abstract class DSSToRelConversionOperation implements ConversionOperation { + + private DSSToRelConversionService conversionService; + + @Override + public void setConversionService(ConversionService conversionService) { + this.conversionService = (DSSToRelConversionService) conversionService; + } + + public DSSToRelConversionService getConversionService() { + return this.conversionService; + } +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ConversionRequestRef.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ConversionRequestRef.java new file mode 100644 index 000000000..c8f510d19 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ConversionRequestRef.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.ref; + +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; + + +public interface ConversionRequestRef extends RequestRef { + + Workspace getWorkspace(); + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/DSSToRelConversionRequestRef.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/DSSToRelConversionRequestRef.java new file mode 100644 index 000000000..f63f90147 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/DSSToRelConversionRequestRef.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.ref; + + +public interface DSSToRelConversionRequestRef extends ConversionRequestRef { + + String getUserName(); + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ProjectToRelConversionRequestRef.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ProjectToRelConversionRequestRef.java new file mode 100644 index 000000000..8b727c0be --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ProjectToRelConversionRequestRef.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestration; +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import java.util.List; + + +public interface ProjectToRelConversionRequestRef extends DSSToRelConversionRequestRef { + + DSSProject getDSSProject(); + + List getDSSOrcList(); + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ProjectToRelConversionRequestRefImpl.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ProjectToRelConversionRequestRefImpl.java new file mode 100644 index 000000000..6a1615ae7 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/ref/ProjectToRelConversionRequestRefImpl.java @@ -0,0 +1,69 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.ref; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestration; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractRequestRef; +import java.util.List; + + +public class ProjectToRelConversionRequestRefImpl extends AbstractRequestRef implements ProjectToRelConversionRequestRef { + + private DSSProject dssProject; + private List dssOrcList; + private Workspace workspace; + private String userName; + + @Override + public DSSProject getDSSProject() { + return dssProject; + } + + public void setDSSProject(DSSProject dssProject) { + this.dssProject = dssProject; + } + + @Override + public List getDSSOrcList() { + return dssOrcList; + } + + public void setDSSOrcList(List dssOrcList) { + this.dssOrcList = dssOrcList; + } + + @Override + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + @Override + public Workspace getWorkspace() { + return workspace; + } + + public void setWorkspace(Workspace workspace) { + this.workspace = workspace; + } + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/AbstractConversionService.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/AbstractConversionService.java new file mode 100644 index 000000000..dcac8295b --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/AbstractConversionService.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.service; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ConversionIntegrationStandard; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.operation.ConversionOperation; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppSingletonIntegrationServiceImpl; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; +import java.util.ArrayList; +import java.util.List; + + +public abstract class AbstractConversionService extends AppSingletonIntegrationServiceImpl + implements ConversionService { + + private List dssLabels = new ArrayList<>(); + private ConversionIntegrationStandard appStandard; + + public void setLabels(List labels) { + dssLabels = labels; + } + + @Override + public List getLabels() { + return dssLabels; + } + + @Override + protected void initOperation(ConversionOperation operation) { + operation.setConversionService(this); + operation.init(); + } + + @Override + public ConversionIntegrationStandard getAppStandard() { + return appStandard; + } + + @Override + public void setAppStandard(ConversionIntegrationStandard appStandard) { + this.appStandard = appStandard; + } +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/ConversionService.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/ConversionService.java new file mode 100644 index 000000000..b03256595 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/ConversionService.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.service; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ConversionIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppIntegrationService; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; +import java.util.List; + + +public interface ConversionService extends AppIntegrationService { + + /** + * Labels by default. + * @return All default labels + */ + List getLabels(); + + void setAppStandard(ConversionIntegrationStandard appStandard); + + ConversionIntegrationStandard getAppStandard(); + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/DSSToRelConversionService.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/DSSToRelConversionService.java new file mode 100644 index 000000000..748b56d01 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/DSSToRelConversionService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.service; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.operation.DSSToRelConversionOperation; + + +public interface DSSToRelConversionService extends ConversionService { + + DSSToRelConversionOperation getDSSToRelConversionOperation(); + + default boolean isConvertAllOrcs() { + return true; + } + +} diff --git a/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/RelToOrchestratorConversionService.java b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/RelToOrchestratorConversionService.java new file mode 100644 index 000000000..9cff6afb0 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-conversion-standard/src/main/java/com/webank/wedatasphere/dss/orchestrator/converter/standard/service/RelToOrchestratorConversionService.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.converter.standard.service; + + +public interface RelToOrchestratorConversionService extends ConversionService { + +} diff --git a/dss-orchestrator/dss-orchestrator-core/pom.xml b/dss-orchestrator/dss-orchestrator-core/pom.xml new file mode 100644 index 000000000..ec531646b --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/pom.xml @@ -0,0 +1,116 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator-core + + + com.webank.wedatasphere.linkis + linkis-module + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-appconn-core + ${dss.version} + + + + linkis-bml-client + + + gson + com.google.code.gson + + + com.webank.wedatasphere.linkis + ${linkis.version} + + + + com.webank.wedatasphere.dss + dss-contextservice + ${dss.version} + + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-orchestrator-common + ${dss.version} + compile + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + + + src/main/java + + **/*.xml + + + + src/main/resources + + **/*.xml + **/*.properties + **/*.yml + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/DSSOrchestrator.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/DSSOrchestrator.java new file mode 100644 index 000000000..899346e23 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/DSSOrchestrator.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import java.util.List; + + + +public interface DSSOrchestrator { + + /** + * 返回Orchestrator的名称，如workflow + * @return + */ + String getName(); + + + /** + * 返回编排关联的AppConn + * @return + */ + AppConn getAppConn(); + + DSSOrchestratorContext getDSSOrchestratorContext(); + + void setAppConn(AppConn appConn); + + /** + *添加当前编排需要使用到在appconn + * @param appconn + */ + void addLinkedAppConn(AppConn appconn); + + + /** + * 为编排提供标签说明，如DEV + * @param dssLabel + */ + void addLinkedDssLabels(DSSLabel dssLabel); + + /** + * 返回所有已经关联到的AppConn + * @return + */ + List getLinkedAppConn(); + + /** + * 用于工具条功能按钮展示，可以查到该模式可以提供的功能按钮 + * @return + */ + List getToolBars(); +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/DSSOrchestratorContext.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/DSSOrchestratorContext.java new file mode 100644 index 000000000..bef0e6cf9 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/DSSOrchestratorContext.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core; + +import com.webank.wedatasphere.dss.common.exception.DSSRuntimeException; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import java.io.Closeable; +import java.util.List; +import java.util.Map; + + +public interface DSSOrchestratorContext extends Closeable { + + void initialize(); + + Map getConfigMap(); + + List getOrchestratorPlugins(); + + default T getDSSOrchestratorPlugin(Class clazz) { + return (T) getOrchestratorPlugins().stream().filter(clazz::isInstance) + .findFirst().orElseThrow(() -> new DSSRuntimeException(50321, "Cannot find " + clazz.getSimpleName())); + } + + boolean isActive(); + +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/exception/DSSOrchestratorErrorException.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/exception/DSSOrchestratorErrorException.java new file mode 100644 index 000000000..601ca0074 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/exception/DSSOrchestratorErrorException.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.exception; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + + +public class DSSOrchestratorErrorException extends DSSErrorException { + public DSSOrchestratorErrorException(int errCode, String desc) { + super(errCode, desc); + } + + public DSSOrchestratorErrorException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/AbstractDSSOrchestratorContext.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/AbstractDSSOrchestratorContext.java new file mode 100644 index 000000000..b2f1799d3 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/AbstractDSSOrchestratorContext.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.impl; + +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import java.util.HashMap; +import java.util.Map; + + +public abstract class AbstractDSSOrchestratorContext implements DSSOrchestratorContext { + + private boolean isClosed = false; + private Map configMap = new HashMap<>(); + + + @Override + public Map getConfigMap() { + return configMap; + } + + @Override + public boolean isActive() { + return !isClosed; + } + + @Override + public void close() { + isClosed = true; + getOrchestratorPlugins().forEach(DSSExceptionUtils.handling(DSSOrchestratorPlugin::close)); + } +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/AbstractOrchestrator.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/AbstractOrchestrator.java new file mode 100644 index 000000000..f11c5f1ee --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/AbstractOrchestrator.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.impl; + +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; + +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext; +import java.util.Arrays; +import java.util.List; + + +abstract class AbstractOrchestrator implements DSSOrchestrator { + + private volatile DSSOrchestratorContext dssOrchestratorContext; + + @Override + public List getToolBars() { + String[] toolNames = {"参数", "资源", "执行", "发布","保存"}; + return Arrays.asList(toolNames); + } + + protected abstract DSSOrchestratorContext createOrchestratorContext(); + + @Override + public DSSOrchestratorContext getDSSOrchestratorContext() { + if(dssOrchestratorContext == null) { + synchronized (this) { + if(dssOrchestratorContext == null) { + dssOrchestratorContext = createOrchestratorContext(); + } + } + } + return dssOrchestratorContext; + } +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/DSSOrchestratorContextImpl.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/DSSOrchestratorContextImpl.java new file mode 100644 index 000000000..6cc3f940d --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/DSSOrchestratorContextImpl.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.impl; + +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import java.util.List; + + +public class DSSOrchestratorContextImpl extends AbstractDSSOrchestratorContext { + + private List plugins; + + @Override + public void initialize() { + plugins = ClassUtils.getInstances(DSSOrchestratorPlugin.class); + plugins.forEach(DSSOrchestratorPlugin::init); + } + + @Override + public List getOrchestratorPlugins() { + return plugins; + } + +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/DefaultOrchestrator.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/DefaultOrchestrator.java new file mode 100644 index 000000000..af8c7bd45 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/impl/DefaultOrchestrator.java @@ -0,0 +1,82 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.impl; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext; +import java.util.ArrayList; +import java.util.List; + + +public class DefaultOrchestrator extends AbstractOrchestrator { + + private static volatile DSSOrchestratorContext orchestratorContext; + + private static void initDSSOrchestratorContext() { + if(orchestratorContext == null) { + synchronized (DefaultOrchestrator.class) { + if(orchestratorContext == null) { + orchestratorContext = new DSSOrchestratorContextImpl(); + orchestratorContext.initialize(); + } + } + } + } + + private List linkedAppConn = new ArrayList<>(); + + private List labels = new ArrayList<>(); + + private AppConn appConn; + + @Override + public void setAppConn(AppConn appConn){ + this.appConn = appConn; + } + + @Override + public String getName() { + return "DefaultOrchestrator"; + } + + @Override + public AppConn getAppConn() { + return this.appConn; + } + + @Override + public void addLinkedAppConn(AppConn appconn) { + linkedAppConn.add(appconn); + } + + @Override + public void addLinkedDssLabels(DSSLabel dssLabel) { + labels.add(dssLabel); + } + + @Override + public List getLinkedAppConn() { + return linkedAppConn; + } + + @Override + protected DSSOrchestratorContext createOrchestratorContext() { + initDSSOrchestratorContext(); + return orchestratorContext; + } +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/plugin/AbstractDSSOrchestratorPlugin.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/plugin/AbstractDSSOrchestratorPlugin.java new file mode 100644 index 000000000..7bff43478 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/plugin/AbstractDSSOrchestratorPlugin.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.plugin; + +import java.io.IOException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class AbstractDSSOrchestratorPlugin implements DSSOrchestratorPlugin { + + protected final Logger LOGGER = LoggerFactory.getLogger(this.getClass()); + + @Override + public void init() { + } + + @Override + public boolean isReady() { + return true; + } + + @Override + public void close() throws IOException { + } +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/plugin/DSSOrchestratorPlugin.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/plugin/DSSOrchestratorPlugin.java new file mode 100644 index 000000000..fde7a4117 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/plugin/DSSOrchestratorPlugin.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.plugin; + +import java.io.Closeable; + + +public interface DSSOrchestratorPlugin extends Closeable { + + void init(); + + boolean isReady(); + +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/type/OrchestratorKindEnum.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/type/OrchestratorKindEnum.java new file mode 100644 index 000000000..8e5aab864 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/type/OrchestratorKindEnum.java @@ -0,0 +1,62 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.type; + + +public enum OrchestratorKindEnum { + + /** + * 编排模式的多种类型 + */ + WORKFLOW(1, "workflow","工作流"), + SINGLE_TASK(2, "singleTask", "单任务"), + COMBINED(3,"combined", "组合编排"); + + private OrchestratorKindEnum(int index, String name, String chName){ + this.index = index; + this.name = name; + this.chName = chName; + } + + private int index; + + private String name; + + private String chName; + + public int getIndex() { + return index; + } + + public String getName() { + return name; + } + + public String getChName() { + return chName; + } + + public static OrchestratorKindEnum getType(int index){ + for (OrchestratorKindEnum type : values()) { + if (type.getIndex() == index) { + return type; + } + } + return WORKFLOW; + } + +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/utils/OrchestratorUtils.java b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/utils/OrchestratorUtils.java new file mode 100644 index 000000000..37120a4dc --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/java/com/webank/wedatasphere/dss/orchestrator/core/utils/OrchestratorUtils.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.utils; + + +public class OrchestratorUtils { + + public static String generateNewVersion() { + return "v000001"; + } + + /** + * 注意： flow版本更新需要同步更新ContextID + * @param oldVersion + * @return + */ + public static String increaseVersion(String oldVersion) { + int num = Integer.parseInt(oldVersion.substring(1)) + 1; + String tmp = "00000" + num; + return "v" + tmp.substring(tmp.length() - 6); + } + + +} diff --git a/dss-orchestrator/dss-orchestrator-core/src/main/scala/com/webank/wedatasphere/dss/orchestrator/core/service/BMLService.scala b/dss-orchestrator/dss-orchestrator-core/src/main/scala/com/webank/wedatasphere/dss/orchestrator/core/service/BMLService.scala new file mode 100644 index 000000000..312b64cfa --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-core/src/main/scala/com/webank/wedatasphere/dss/orchestrator/core/service/BMLService.scala @@ -0,0 +1,183 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.core.service + +import java.io.{ByteArrayInputStream, InputStream} +import java.util +import java.util.UUID + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException +import com.webank.wedatasphere.dss.common.utils.IoUtils +import com.webank.wedatasphere.linkis.bml.client.{BmlClient, BmlClientFactory} +import com.webank.wedatasphere.linkis.bml.protocol.{BmlDownloadResponse, BmlUpdateResponse, BmlUploadResponse} +import com.webank.wedatasphere.linkis.common.utils.{JavaLog, Utils} +import org.apache.commons.io.IOUtils +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ + + +@Component +class BMLService extends JavaLog{ + + def upload(userName: String, content: String, fileName: String, projectName:String): util.Map[String, Object] = { + val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) + val client: BmlClient = createBMLClient(userName) + val resource: BmlUploadResponse = client.uploadShareResource(userName, projectName, fileName, inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def upload(userName: String, inputStream: InputStream, fileName: String, projectName:String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + val resource: BmlUploadResponse = client.uploadShareResource(userName, projectName, fileName, inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def update(userName: String, resourceId: String, inputStream: InputStream): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + val resource: BmlUpdateResponse = client.updateShareResource(userName, resourceId, "", inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def update(userName: String, resourceId: String, content: String): util.Map[String, Object] = { + val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) + val client: BmlClient = createBMLClient(userName) + val resource: BmlUpdateResponse = client.updateShareResource(userName, resourceId, UUID.randomUUID().toString+".json", inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def query(userName: String, resourceId: String, version: String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + var resource: BmlDownloadResponse = null + if (version == null) { + resource = client.downloadShareResource(userName, resourceId) + } else { + resource = client.downloadShareResource(userName, resourceId, version) + } + if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") + val map = new util.HashMap[String, Object] + map += "path" -> resource.fullFilePath + map += "string" -> inputstremToString(resource.inputStream) + } + + def download(userName: String, resourceId: String, version: String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + var resource: BmlDownloadResponse = null + if (version == null) { + resource = client.downloadShareResource(userName, resourceId) + } else { + resource = client.downloadShareResource(userName, resourceId, version) + } + if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") + val map = new util.HashMap[String, Object] + map += "path" -> resource.fullFilePath + map += "is" -> resource.inputStream + } + + def downloadToLocalPath(userName: String, resourceId: String, version: String, path: String): String = { + val result = download(userName,resourceId,version) + val is = result.get("is").asInstanceOf[InputStream] + val os = IoUtils.generateExportOutputStream(path) + Utils.tryFinally(IOUtils.copy(is,os)){ + IOUtils.closeQuietly(os) + IOUtils.closeQuietly(is) + } + path + } + + def downloadAndGetFlowJson(userName: String, resourceId: String, version: String, path: String): String = { + downloadToLocalPath(userName,resourceId,version,path) + //因为下载到指定目录后返回的resource的Stream为null,只能从文件重新读取。 + val is = IoUtils.generateInputInputStream(path) + Utils.tryFinally(inputstremToString(is))(IOUtils.closeQuietly(is)) + } + + def readLocalResourceFile(userName: String,readPath: String): InputStream ={ + IoUtils.generateInputInputStream(readPath) + } + + def readLocalFlowJsonFile(userName: String,readPath: String): String ={ + var inputStream:InputStream = null + var flowJson:String = null + Utils.tryFinally{ + inputStream = IoUtils.generateInputInputStream(readPath) + flowJson = inputstremToString(inputStream) + }{ + IOUtils.closeQuietly(inputStream) + } + flowJson + } + + private def inputstremToString(inputStream: InputStream): String = { + scala.io.Source.fromInputStream(inputStream).mkString + } + + private def createBMLClient(userName: String): BmlClient = { + if (userName == null) + BmlClientFactory.createBmlClient() + else + BmlClientFactory.createBmlClient(userName) + } + + def createBmlProject(username:String, projectName:String, editUsers:util.List[String], + accessUsers:util.List[String] ): Unit ={ + val client = createBMLClient(username) + val response = client.createBmlProject(username, projectName, accessUsers, editUsers) + if (response.isSuccess){ + logger.info(s"for user $username create bml project $projectName success") + }else{ + logger.error(s"for user $username create bml project $projectName failed") + } + } + + def attachResourceAndProject(username:String, projectName:String, resourceId:String):Unit = { + val client = createBMLClient(username) + val response = client.attachResourceAndProject(projectName, resourceId) + if (response.isSuccess){ + logger.info(s"attach $username and $projectName success") + }else{ + logger.error(s"attach $username and $projectName failed") + } + } + + + def updateProjectPriv(projectName:String, username:String, editUsers:util.List[String], + accessUsers:util.List[String]): Unit ={ + val client = createBMLClient(username) + val response = client.updateProjectPriv(username, projectName, editUsers, accessUsers) + if (response.isSuccess){ + logger.info(s"attach $username and $projectName success") + }else{ + logger.error(s"attach $username and $projectName failed") + } + } + + + +} diff --git a/dss-orchestrator/dss-orchestrator-db/pom.xml b/dss-orchestrator/dss-orchestrator-db/pom.xml new file mode 100644 index 000000000..d06951494 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-db/pom.xml @@ -0,0 +1,77 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator-db + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + + + src/main/java + + **/*.xml + + + + src/main/resources + + **/*.xml + **/*.properties + **/*.yml + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/dss-orchestrator-db/src/main/java/com/webank/wedatasphere/dss/orchestrator/db/dao/OrchestratorMapper.java b/dss-orchestrator/dss-orchestrator-db/src/main/java/com/webank/wedatasphere/dss/orchestrator/db/dao/OrchestratorMapper.java new file mode 100644 index 000000000..5a480fb65 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-db/src/main/java/com/webank/wedatasphere/dss/orchestrator/db/dao/OrchestratorMapper.java @@ -0,0 +1,127 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.db.dao; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorVersion; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.common.entity.OrchestratorUser; +import org.apache.ibatis.annotations.*; + +import java.util.Date; +import java.util.List; + +@Mapper +public interface OrchestratorMapper { + + void addOrchestrator(DSSOrchestratorInfo dssOrchestratorInfo); + + DSSOrchestratorInfo getOrchestrator(Long id); + + DSSOrchestratorInfo getOrchestratorByUUID(String uuid); + + void updateOrchestrator(DSSOrchestratorInfo dssOrchestratorInfo); + + void deleteOrchestrator(Long id); + + /** + * @param dssOrchestratorVersion + */ + + void addOrchestratorVersion(DSSOrchestratorVersion dssOrchestratorVersion); + + DSSOrchestratorVersion getOrchestratorVersion(Long versionId); + + /** + * 根据id查找最新的版本信息 + * + * @param orchestratorId + * @return + */ + DSSOrchestratorVersion getLatestOrchestratorVersionById(Long orchestratorId); + + void updateOrchestratorVersion(DSSOrchestratorVersion dssOrchestratorVersion); + + void deleteOrchestratorVersion(Long versionId); + + /** + * 根据id查找所有的版本信息 + * + * @param orchestratorId + * @return + * + */ + List getVersionByOrchestratorId(Long orchestratorId); + + OrchestratorInfo getOrcInfoByAppId(@Param("appId")Long appId); + + @Select("select max(id) from dss_orchestrator_version_info where `orchestrator_id` = #{orchestratorId}") + Long findLatestOrcVersionId(@Param("orchestratorId") Long orchestratorId); + + List getOrchestratorVersions(@Param("projectId") Long projectId, @Param("orchestratorId") Long orchestratorId); + + @Select("select max(`version`) from dss_orchestrator_version_info where orchestrator_id = #{orchestratorId}") + String getLatestVersion(@Param("orchestratorId") Long orchestratorId); + + + @Select("select uuid from dss_orchestrator_info where project_id = #{projectId} and name = #{name}") + String getOrcNameByParam(@Param("projectId") Long projectId,@Param("name") String name); + + @Select("select id from `dss_orchestrator_info` where `project_id` = #{projectId} and `is_published` = 1") + List getAllOrcIdsByProjectId(@Param("projectId")Long projectId); + + @Select("select max(`app_id`) from `dss_orchestrator_version_info` where `orchestrator_id` = #{orchestratorId} and `version` = #{version}") + Long getAppIdByVersion(@Param("orchestratorId") Long orchestratorId, @Param("version") String version); + + @Update("update `dss_orchestrator_info` set `is_published` = 1 where id = #{orchestratorId}") + void setPublished(@Param("orchestratorId")Long orchestratorId); + + @Select("select `name` from dss_orchestrator_info where `id` = #{orchestratorId}") + String getOrchestratorNameById(@Param("orchestratorId") int orchestratorId); + + @Delete("delete from `dss_orchestrator_schedule_info` where `orchestrator_id` = #{orchestratorId}") + int deleteScheduleInfo(@Param("orchestratorId") Integer orchestratorId); + + @Update("update dss_orchestrator_schedule_info set active_flag = #{activeFlag} where `orchestrator_id` = #{orchestratorId}") + int updateScheduleInfoActiveFlag(@Param("orchestratorId") Long orchestratorId,@Param("activeFlag") String activeFlag ); + + + @Insert({ + "" + }) + void setOrchestratorPriv(@Param("workspaceId")int workspaceId, + @Param("projectId")Long projectId, @Param("orchestratorId")int orchestratorId, + @Param("accessUsers")List accessUsers, @Param("priv")int priv, @Param("updateTime") Date date); + + @Delete("delete from `dss_orchestrator_user` " + + "where `workspace_id` = #{workspaceId} " + + "and `project_id` = #{projectId} " + + "and `orchestrator_id` = #{orchestratorId}") + void deleteAllOrchestratorPriv(@Param("workspaceId")int workspaceId, @Param("projectId")Long projectId, @Param("orchestratorId")int orchestratorId); + + + List getOrchestratorUserByOrcId(@Param("orchestratorId") Long orchestratorId); + + +} diff --git a/dss-orchestrator/dss-orchestrator-db/src/main/java/com/webank/wedatasphere/dss/orchestrator/db/dao/impl/orchestratorMapper.xml b/dss-orchestrator/dss-orchestrator-db/src/main/java/com/webank/wedatasphere/dss/orchestrator/db/dao/impl/orchestratorMapper.xml new file mode 100644 index 000000000..e7b603266 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-db/src/main/java/com/webank/wedatasphere/dss/orchestrator/db/dao/impl/orchestratorMapper.xml @@ -0,0 +1,176 @@ + + + + + + + + + + + + + id,`name`,`type`,`desc`,`creator`,`create_time`,`project_id`,`uses`,`appconn_name`,`uuid`,`secondary_type` + + + + id,`orchestrator_id`,`app_id`,`source`,`version`,`comment`,`update_time`,`updater`,`project_id`,`content`,`context_id` + + + + + + + + INSERT INTO dss_orchestrator_info () + VALUES + (#{id},#{name},#{type},#{desc},#{creator},#{createTime},#{projectId},#{uses},#{appConnName},#{uuid},#{secondaryType}) + + + + UPDATE dss_orchestrator_info + + `name`=#{name}, + `desc`=#{desc}, + `appconn_name`=#{appConnName}, + `uses`=#{uses}, + `secondary_type`=#{secondaryType} + + WHERE id = #{id} + + + + + DELETE + FROM + dss_orchestrator_info + WHERE id = #{id} + + + + INSERT INTO dss_orchestrator_version_info () + VALUES + (#{id},#{orchestratorId},#{appId},#{source},#{version},#{comment},#{updateTime},#{updater},#{projectId},#{content},#{contextId}) + + + + + UPDATE dss_orchestrator_version_info + + `source` = #{source}, + `app_id` = #{appId}, + `content`= #{content}, + `comment` = #{comment}, + `context_id` = #{contextId}, + `update_time` = #{updateTime}, + `updater` = #{updater} + + WHERE id = #{id} + + + + + + + + + + DELETE + FROM + dss_orchestrator_version_info + WHERE id = #{versionId} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/dss-orchestrator-loader/pom.xml b/dss-orchestrator/dss-orchestrator-loader/pom.xml new file mode 100644 index 000000000..4510262b0 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/pom.xml @@ -0,0 +1,71 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator-loader + + + jakarta.annotation + jakarta.annotation-api + 1.3.5 + provided + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-orchestrator-core + ${dss.version} + + + com.webank.wedatasphere.dss + dss-orchestrator-db + ${dss.version} + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + org.springframework + spring-context + ${spring.version} + provided + + + com.webank.wedatasphere.dss + dss-appconn-manager-core + ${dss.version} + + + + \ No newline at end of file diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/DefaultLinkedAppConnResolver.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/DefaultLinkedAppConnResolver.java new file mode 100644 index 000000000..ac5332c70 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/DefaultLinkedAppConnResolver.java @@ -0,0 +1,51 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import java.util.ArrayList; +import java.util.List; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + + + +@Component +class DefaultLinkedAppConnResolver implements LinkedAppConnResolver { + + + private static final Logger LOGGER = LoggerFactory.getLogger(DefaultLinkedAppConnResolver.class); + + + static{ + LOGGER.info("component resolver inited"); + } + + @Override + public List resolveAppConnByUser(String userName, String workspaceName, String typeName) { + //todo 后面可以使用数据库表来定义用户可以加载的AppConn. + List appConns = new ArrayList<>(); + for(AppConn appConn : AppConnManager.getAppConnManager().listAppConns()){ + //可以在这里根据用户情况和工作空间情况，限制appConn的加载 + appConns.add(appConn); + } + + return appConns; + } +} diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/DefaultOrchestratorLoader.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/DefaultOrchestratorLoader.java new file mode 100644 index 000000000..02da54079 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/DefaultOrchestratorLoader.java @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnErrorException; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestratorContext; +import com.webank.wedatasphere.dss.orchestrator.core.impl.DefaultOrchestrator; +import java.util.List; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +@Component +public class DefaultOrchestratorLoader implements OrchestratorLoader { + private static final Logger LOGGER = LoggerFactory.getLogger(DefaultOrchestratorLoader.class); + @Autowired + private DSSOrchestratorContext dssOrchestratorContext; + + @Autowired + private LinkedAppConnResolver linkedAppConnResolver; + + + @Override + public DSSOrchestrator loadOrchestrator(String userName, + String workspaceName, + String typeName, + String appConnName, + List dssLabels) throws AppConnErrorException { + + //todo load DSSOrchestatror by type name + DSSOrchestrator dssOrchestrator = new DefaultOrchestrator() { + @Override + protected DSSOrchestratorContext createOrchestratorContext() { + return dssOrchestratorContext; + } + }; + + //向工作流添加实现了第三级规范的AppConn + List appConnList = linkedAppConnResolver.resolveAppConnByUser(userName, workspaceName, typeName); + for (AppConn appConn : appConnList) { + if(appConn instanceof OnlyDevelopmentAppConn){ + dssOrchestrator.addLinkedAppConn(appConn); + } + + } + LOGGER.info("Load dss orchestrator:"+appConnName+",typeName:"+typeName); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(appConnName); + dssLabels.forEach(dssOrchestrator::addLinkedDssLabels); + dssOrchestrator.setAppConn(appConn); + return dssOrchestrator; + } +} diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/LinkedAppConnResolver.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/LinkedAppConnResolver.java new file mode 100644 index 000000000..66bc4c748 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/LinkedAppConnResolver.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; + + +import java.util.List; + +public interface LinkedAppConnResolver { + List resolveAppConnByUser(String userName, String workspaceName, String typeName); +} diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/OrchestratorLoader.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/OrchestratorLoader.java new file mode 100644 index 000000000..382e6533e --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/OrchestratorLoader.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader; + + +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; + +import java.util.List; + +public interface OrchestratorLoader { + + /** + * 用于返回一个指定类型的的Orchestrator + * @param userName + * @param workspaceName + * @param typeName + * @param appConnName 唯一标识一种类型的AppConn,比如workflowOrchestratorAppConn + * @return + */ + DSSOrchestrator loadOrchestrator(String userName, + String workspaceName, + String typeName, + String appConnName, + List dssLabels) throws AppConnErrorException; +} diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/OrchestratorManager.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/OrchestratorManager.java new file mode 100644 index 000000000..02304da1c --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/OrchestratorManager.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader; + +import com.webank.wedatasphere.dss.appconn.core.exception.AppConnErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + + +@Component +public class OrchestratorManager { + + private final static Logger logger = LoggerFactory.getLogger(OrchestratorManager.class); + + private Map cacheDssOrchestrator = new ConcurrentHashMap<>(); + + @Autowired + private DefaultOrchestratorLoader defaultOrchestratorLoader; + + public DSSOrchestrator getOrCreateOrchestrator(String userName, + String workspaceName, + String typeName, + String appConnName, + List dssLabels) { + String findKey = getCacheKey(userName, workspaceName, typeName, appConnName); + DSSOrchestrator dssOrchestrator = cacheDssOrchestrator.get(findKey); + if (null == dssOrchestrator) { + try { + + dssOrchestrator = defaultOrchestratorLoader.loadOrchestrator(userName, workspaceName, typeName, appConnName, dssLabels); + + cacheDssOrchestrator.put(findKey, dssOrchestrator); + } catch (AppConnErrorException e) { + logger.error("OrchestratorManager get DSSOrchestrator exception!", e); + } + } + return dssOrchestrator; + } + + protected String getCacheKey(String userName, String workspaceName, String typeName, String appConnName) { + return userName + "_" + workspaceName + "_" + typeName + "_" + appConnName; + } +} diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/SpringDSSOrchestratorContext.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/SpringDSSOrchestratorContext.java new file mode 100644 index 000000000..d1591764c --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/SpringDSSOrchestratorContext.java @@ -0,0 +1,66 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader; + +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.core.impl.AbstractDSSOrchestratorContext; +import com.webank.wedatasphere.dss.orchestrator.core.plugin.DSSOrchestratorPlugin; +import java.util.List; +import java.util.stream.Collectors; +import javax.annotation.PostConstruct; +import javax.annotation.PreDestroy; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + + +@Component +public class SpringDSSOrchestratorContext extends AbstractDSSOrchestratorContext { + + private final Logger LOGGER = LoggerFactory.getLogger(SpringDSSOrchestratorContext.class); + + @Autowired + private List dssOrchestratorPlugins; + + @Override + @PostConstruct + public void initialize() { + if(dssOrchestratorPlugins == null || dssOrchestratorPlugins.isEmpty()) { + dssOrchestratorPlugins = ClassUtils.getInstances(DSSOrchestratorPlugin.class); + } else { + List others = ClassUtils.getClasses(DSSOrchestratorPlugin.class).stream() + .filter(clazz -> dssOrchestratorPlugins.stream().noneMatch(clazz::isInstance)) + .map(DSSExceptionUtils.map(Class::newInstance)).collect(Collectors.toList()); + dssOrchestratorPlugins.addAll(others); + } + dssOrchestratorPlugins.forEach(DSSOrchestratorPlugin::init); + LOGGER.info("The DSSOrchestratorPlugin list is {}.", dssOrchestratorPlugins); + } + + @Override + public List getOrchestratorPlugins() { + return dssOrchestratorPlugins; + } + + @Override + @PreDestroy + public void close() { + super.close(); + } +} diff --git a/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/utils/OrchestratorLoaderUtils.java b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/utils/OrchestratorLoaderUtils.java new file mode 100644 index 000000000..051763995 --- /dev/null +++ b/dss-orchestrator/dss-orchestrator-loader/src/main/java/com/webank/wedatasphere/dss/orchestrator/loader/utils/OrchestratorLoaderUtils.java @@ -0,0 +1,80 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.orchestrator.loader.utils; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestratorInfo; +import com.webank.wedatasphere.dss.orchestrator.core.DSSOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.loader.OrchestratorManager; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.exception.NoSuchAppInstanceException; +import javafx.util.Pair; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; +import java.util.List; + + +@Component +public class OrchestratorLoaderUtils { + private static final Logger LOGGER = LoggerFactory.getLogger(OrchestratorLoaderUtils.class); + @Autowired + private OrchestratorManager orchestratorManager; + private static OrchestratorLoaderUtils orchestratorLoaderUtils; + + @PostConstruct + public void init(){ + orchestratorLoaderUtils = this; + orchestratorLoaderUtils.orchestratorManager=this.orchestratorManager; + } + + + public static Pair getOrcDevelopStandard(String userName, + String workspaceName, + DSSOrchestratorInfo dssOrchestratorInfo, + List dssLabels) throws NoSuchAppInstanceException { + DSSOrchestrator dssOrchestrator = orchestratorLoaderUtils.orchestratorManager.getOrCreateOrchestrator(userName, + workspaceName, dssOrchestratorInfo.getType(), dssOrchestratorInfo.getAppConnName(), dssLabels); + if (null != dssOrchestrator) { + AppConn orchestratorAppConn = dssOrchestrator.getAppConn(); + + if(orchestratorAppConn instanceof OnlyDevelopmentAppConn) { + DevelopmentIntegrationStandard developmentIntegrationStandard = ((OnlyDevelopmentAppConn) orchestratorAppConn).getOrCreateDevelopmentStandard(); + + //todo labels判别 + List appInstance = orchestratorAppConn.getAppDesc().getAppInstancesByLabels(dssLabels); + if (appInstance.size() > 0 && null != developmentIntegrationStandard) { + return new Pair(appInstance.get(0),developmentIntegrationStandard); + + }else{ + return null; + } + } + } else { + LOGGER.error("Can not get dssOrchestrator from manager"); + return null; + } + + return null; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/pom.xml new file mode 100644 index 000000000..ab6e7e2bf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/pom.xml @@ -0,0 +1,226 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-flow-execution-server + + + + + com.webank.wedatasphere.linkis + linkis-jobhistory + ${linkis.version} + + + com.webank.wedatasphere.linkis + linkis-entrance + ${linkis.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + spring-tx + org.springframework + + + commons-text + org.apache.commons + + + json4s-jackson_2.11 + org.json4s + + + linkis-label-common + com.webank.wedatasphere.linkis + + + + + + com.webank.wedatasphere.dss + dss-linkis-node-execution + ${dss.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + commons-beanutils + commons-beanutils + + + linkis-label-common + com.webank.wedatasphere.linkis + + + + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + org.springframework.cloud + spring-cloud-starter-netflix-eureka-client + + + javax.validation + validation-api + + + org.glassfish.hk2.external + bean-validator + + + + + + com.webank.wedatasphere.linkis + linkis-storage + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-workflow-sdk + ${dss.version} + + + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + + org.apache.commons + commons-lang3 + ${commons.lang3.version} + provided + + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.2 + + 1.8 + 1.8 + UTF-8 + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + -target:jvm-1.8 + + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + + + + + ${project.artifactId}-${project.version} + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/assembly/distribution.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..d14b0412b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/assembly/distribution.xml @@ -0,0 +1,43 @@ + + + + dss-framework-execution-server + + dir + + true + dss-flow-execution-server + + + + + + lib + true + true + false + true + true + + + + + + diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/dao/TaskMapper.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/dao/TaskMapper.java new file mode 100644 index 000000000..af84a34ce --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/dao/TaskMapper.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.dao; + +import com.webank.wedatasphere.dss.flow.execution.entrance.entity.WorkflowQueryTask; +import org.apache.ibatis.annotations.Param; +import java.util.Date; +import java.util.List; + + +public interface TaskMapper { + + List selectTask(WorkflowQueryTask queryTask); + + void insertTask(WorkflowQueryTask queryTask); + + void updateTask(WorkflowQueryTask queryTask); + + List search(@Param("taskID") Long taskID, @Param("umUser") String username, @Param("status") List status, + @Param("startDate") Date startDate, @Param("endDate") Date endDate, @Param("executeApplicationName") String executeApplicationName, + @Param("instance") String instance, @Param("execId") String execId); + + String selectTaskStatusForUpdate(Long taskID); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/dao/impl/taskMapper.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/dao/impl/taskMapper.xml new file mode 100644 index 000000000..d43e9763f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/dao/impl/taskMapper.xml @@ -0,0 +1,123 @@ + + + + + + + + + + + + + + + + id, `instance`, `exec_id`, `um_user`, `execution_code`, `progress`,`log_path`, + `result_location`,`status`,`created_time`,`updated_time`,`run_type`,`err_code`,`err_desc`,`execute_application_name`,`request_application_name`,`script_path`,`params`,`engine_instance`, + `engine_start_time`, `task_resource`, `submit_user` , `label_json` + + + + INSERT INTO dss_workflow_task() + VALUES (#{taskID},#{instance},#{execId}, + #{umUser},#{executionCode},#{progress}, + #{logPath},#{resultLocation},#{status}, + #{createdTime},#{updatedTime},#{runType}, + #{errCode},#{errDesc},#{executeApplicationName}, + #{requestApplicationName},#{sourceJson},#{paramsJson}, + #{engineInstance},#{engineStartTime}, #{taskResource}, #{submitUser}, #{labelJson}) + + + + + + + + + UPDATE dss_workflow_task + + instance=#{instance}, + exec_id=#{execId}, + um_user=#{umUser}, + execution_code=#{executionCode}, + progress=#{progress}, + log_path=#{logPath}, + result_location=#{resultLocation}, + status=#{status}, + created_time=#{createdTime}, + updated_time=#{updatedTime}, + run_type=#{runType}, + err_code=#{errCode}, + err_desc=#{errDesc}, + execute_application_name=#{executeApplicationName}, + request_application_name=#{requestApplicationName}, + script_path = #{sourceJson}, + params = #{paramsJson}, + engine_instance = #{engineInstance}, + engine_start_time = #{engineStartTime}, + task_resource = #{taskResource} + + WHERE id =#{taskID} + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/WorkflowQueryTask.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/WorkflowQueryTask.java new file mode 100644 index 000000000..cbd080812 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/WorkflowQueryTask.java @@ -0,0 +1,241 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.entity; + +import java.util.Date; + + +public class WorkflowQueryTask { + private Long taskID; + private String instance; + private String execId; + private String umUser; + private String engineInstance; + private String executionCode; + private Float progress; + private String logPath; + private String resultLocation; + private String status; + private Date createdTime; + private Date updatedTime; + private String engineType; + private Integer errCode; + private String errDesc; + private String executeApplicationName; + private String requestApplicationName; + private String runType; + private String paramsJson; + private String sourceJson; + private Date engineStartTime; + private String taskResource; + + private String submitUser; + + private String labelJson; + + public Date getEngineStartTime() { + return engineStartTime; + } + + public void setEngineStartTime(Date engineStartTime) { + this.engineStartTime = engineStartTime; + } + + public String getSourceJson() { + return sourceJson; + } + + public void setSourceJson(String sourceJson) { + this.sourceJson = sourceJson; + } + + public Long getTaskID() { + return taskID; + } + + public void setTaskID(Long taskID) { + this.taskID = taskID; + } + + public String getInstance() { + return instance; + } + + public void setInstance(String instance) { + this.instance = instance; + } + + public String getExecId() { + return execId; + } + + public void setExecId(String execId) { + this.execId = execId; + } + + public String getUmUser() { + return umUser; + } + + public void setUmUser(String umUser) { + this.umUser = umUser; + } + + public String getEngineInstance() { + return engineInstance; + } + + public void setEngineInstance(String engineInstance) { + this.engineInstance = engineInstance; + } + + public String getExecutionCode() { + return executionCode; + } + + public void setExecutionCode(String executionCode) { + this.executionCode = executionCode; + } + + public Float getProgress() { + return progress; + } + + public void setProgress(Float progress) { + this.progress = progress; + } + + public String getLogPath() { + return logPath; + } + + public void setLogPath(String logPath) { + this.logPath = logPath; + } + + public String getResultLocation() { + return resultLocation; + } + + public void setResultLocation(String resultLocation) { + this.resultLocation = resultLocation; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public Date getCreatedTime() { + return createdTime; + } + + public void setCreatedTime(Date createdTime) { + this.createdTime = createdTime; + } + + public Date getUpdatedTime() { + return updatedTime; + } + + public void setUpdatedTime(Date updatedTime) { + this.updatedTime = updatedTime; + } + + public String getEngineType() { + return engineType; + } + + public void setEngineType(String engineType) { + this.engineType = engineType; + } + + public Integer getErrCode() { + return errCode; + } + + public void setErrCode(Integer errCode) { + this.errCode = errCode; + } + + public String getErrDesc() { + return errDesc; + } + + public void setErrDesc(String errDesc) { + this.errDesc = errDesc; + } + + public String getExecuteApplicationName() { + return executeApplicationName; + } + + public void setExecuteApplicationName(String executeApplicationName) { + this.executeApplicationName = executeApplicationName; + } + + public String getRequestApplicationName() { + return requestApplicationName; + } + + public void setRequestApplicationName(String requestApplicationName) { + this.requestApplicationName = requestApplicationName; + } + + public String getRunType() { + return runType; + } + + public void setRunType(String runType) { + this.runType = runType; + } + + public String getParamsJson() { + return paramsJson; + } + + public void setParamsJson(String paramsJson) { + this.paramsJson = paramsJson; + } + + public String getTaskResource() { + return taskResource; + } + + public void setTaskResource(String taskResource) { + this.taskResource = taskResource; + } + + public String getSubmitUser() { + return submitUser; + } + + public void setSubmitUser(String submitUser) { + this.submitUser = submitUser; + } + + public String getLabelJson() { + return labelJson; + } + + public void setLabelJson(String labelJson) { + this.labelJson = labelJson; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/WorkflowQueryTaskVO.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/WorkflowQueryTaskVO.java new file mode 100644 index 000000000..3fc5dfba2 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/entity/WorkflowQueryTaskVO.java @@ -0,0 +1,243 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.entity; + +import java.util.Date; + + +public class WorkflowQueryTaskVO { + private Long taskID; + private String instance; + private String execId; + private String umUser; + private String engineInstance; + private String executionCode; + private Float progress; + private String logPath; + private String resultLocation; + private String status; + private Date createdTime; + private Date updatedTime; + private String engineType; + private Integer errCode; + private String errDesc; + private String executeApplicationName; + private String requestApplicationName; + private String runType; + private String paramsJson; + private Long costTime; + private String strongerExecId; + private String sourceJson; + /** + * source字段:用来将sourceJson的value取出来进行拼接返回给前台展示 + */ + private String sourceTailor; + + private Date engineStartTime; + + public Date getEngineStartTime() { + return engineStartTime; + } + + public void setEngineStartTime(Date engineStartTime) { + this.engineStartTime = engineStartTime; + } + + public String getSourceTailor() { + return sourceTailor; + } + + public void setSourceTailor(String sourceTailor) { + this.sourceTailor = sourceTailor; + } + + public String getSourceJson() { + return sourceJson; + } + + public void setSourceJson(String sourceJson) { + this.sourceJson = sourceJson; + } + + public Long getTaskID() { + return taskID; + } + + public void setTaskID(Long taskID) { + this.taskID = taskID; + } + + public String getInstance() { + return instance; + } + + public void setInstance(String instance) { + this.instance = instance; + } + + public String getExecId() { + return execId; + } + + public void setExecId(String execId) { + this.execId = execId; + } + + public String getUmUser() { + return umUser; + } + + public void setUmUser(String umUser) { + this.umUser = umUser; + } + + public String getEngineInstance() { + return engineInstance; + } + + public void setEngineInstance(String engineInstance) { + this.engineInstance = engineInstance; + } + + public String getExecutionCode() { + return executionCode; + } + + public void setExecutionCode(String executionCode) { + this.executionCode = executionCode; + } + + public Float getProgress() { + return progress; + } + + public void setProgress(Float progress) { + this.progress = progress; + } + + public String getLogPath() { + return logPath; + } + + public void setLogPath(String logPath) { + this.logPath = logPath; + } + + public String getResultLocation() { + return resultLocation; + } + + public void setResultLocation(String resultLocation) { + this.resultLocation = resultLocation; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public Date getCreatedTime() { + return createdTime; + } + + public void setCreatedTime(Date createdTime) { + this.createdTime = createdTime; + } + + public Date getUpdatedTime() { + return updatedTime; + } + + public void setUpdatedTime(Date updatedTime) { + this.updatedTime = updatedTime; + } + + public String getEngineType() { + return engineType; + } + + public void setEngineType(String engineType) { + this.engineType = engineType; + } + + public Integer getErrCode() { + return errCode; + } + + public void setErrCode(Integer errCode) { + this.errCode = errCode; + } + + public String getErrDesc() { + return errDesc; + } + + public void setErrDesc(String errDesc) { + this.errDesc = errDesc; + } + + public String getExecuteApplicationName() { + return executeApplicationName; + } + + public void setExecuteApplicationName(String executeApplicationName) { + this.executeApplicationName = executeApplicationName; + } + + public String getRequestApplicationName() { + return requestApplicationName; + } + + public void setRequestApplicationName(String requestApplicationName) { + this.requestApplicationName = requestApplicationName; + } + + public String getRunType() { + return runType; + } + + public void setRunType(String runType) { + this.runType = runType; + } + + public String getParamsJson() { + return paramsJson; + } + + public void setParamsJson(String paramsJson) { + this.paramsJson = paramsJson; + } + + public Long getCostTime() { + return costTime; + } + + public void setCostTime(Long costTime) { + this.costTime = costTime; + } + + public String getStrongerExecId() { + return strongerExecId; + } + + public void setStrongerExecId(String strongerExecId) { + this.strongerExecId = strongerExecId; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppConnLinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppConnLinkisJob.java new file mode 100644 index 000000000..96b93c2a5 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionAppConnLinkisJob.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.job; + +import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.job.AbstractAppConnLinkisJob; + + +public class FlowExecutionAppConnLinkisJob extends AbstractAppConnLinkisJob { + + + @Override + public String getSubmitUser() { + return getJobProps().get(FlowExecutionEntranceConfiguration.FLOW_SUBMIT_USER()); + } + + + + @Override + public String getUser() { + return getSubmitUser(); + //return getJobProps().get(FlowExecutionEntranceConfiguration.PROXY_USER()); + } + + @Override + public String getJobName() { + return getJobProps().get(FlowExecutionEntranceConfiguration.JOB_ID()); + } + + +} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisJob.java similarity index 88% rename from dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisJob.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisJob.java index bd27ae43c..80a2a528d 100644 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisJob.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionCommonLinkisJob.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -28,9 +27,7 @@ import java.util.ArrayList; import java.util.Map; -/** - * Created by peacewong on 2019/11/3. - */ + public class FlowExecutionCommonLinkisJob extends AbstractCommonLinkisJob { @@ -42,7 +39,8 @@ public String getSubmitUser() { @Override public String getUser() { - return getJobProps().get(FlowExecutionEntranceConfiguration.PROXY_USER()); + return getSubmitUser(); + //return getJobProps().get(FlowExecutionEntranceConfiguration.PROXY_USER()); } @Override diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/persistence/WorkflowPersistenceEngine.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/persistence/WorkflowPersistenceEngine.java new file mode 100644 index 000000000..5d52b601b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/persistence/WorkflowPersistenceEngine.java @@ -0,0 +1,228 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.flow.execution.entrance.persistence; + +import com.webank.wedatasphere.dss.flow.execution.entrance.service.WorkflowQueryService; +import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils; +import com.webank.wedatasphere.linkis.common.exception.ErrorException; +import com.webank.wedatasphere.linkis.entrance.exception.EntranceIllegalParamException; +import com.webank.wedatasphere.linkis.entrance.exception.EntranceRPCException; +import com.webank.wedatasphere.linkis.entrance.exception.QueryFailedException; +import com.webank.wedatasphere.linkis.entrance.persistence.AbstractPersistenceEngine; +import com.webank.wedatasphere.linkis.governance.common.entity.job.JobRequest; +import com.webank.wedatasphere.linkis.governance.common.entity.job.SubJobDetail; +import com.webank.wedatasphere.linkis.governance.common.entity.job.SubJobInfo; +import com.webank.wedatasphere.linkis.governance.common.entity.task.*; +import com.webank.wedatasphere.linkis.protocol.constants.TaskConstant; +import com.webank.wedatasphere.linkis.protocol.task.Task; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.BeanUtils; +import org.springframework.beans.factory.annotation.Autowired; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + + +public class WorkflowPersistenceEngine extends AbstractPersistenceEngine { + + @Autowired + private WorkflowQueryService workflowQueryService; + private static final Logger logger = LoggerFactory.getLogger(WorkflowPersistenceEngine.class); + + { + logger.info("WorkflowPersistenceEngine Registered"); + } + + public WorkflowPersistenceEngine() { + + } + + + @Override + public void persist(JobRequest jobRequest) throws QueryFailedException, EntranceIllegalParamException, EntranceRPCException { + + if (jobRequest == null) { + throw new EntranceIllegalParamException(20004, "task can not be null, unable to do persist operation"); + } + RequestInsertTask requestInsertTask = new RequestInsertTask(); + RequestPersistTask requestPersistTask = FlowExecutionUtils.jobRequest2RequestPersistTask(jobRequest); + BeanUtils.copyProperties(requestPersistTask, requestInsertTask); + ResponsePersist responsePersist = null; + + try { + responsePersist = workflowQueryService.add(requestInsertTask); + } catch (Exception e) { + throw new EntranceRPCException(20020, "sender rpc failed", e); + } + + if (responsePersist != null) { + int status = responsePersist.getStatus(); + String message = responsePersist.getMsg(); + if (status != 0) { + throw new QueryFailedException(20011, "insert task failed, reason: " + message); + } + Map data = responsePersist.getData(); + Object object = data.get(TaskConstant.TASKID); + if (object == null) { + throw new QueryFailedException(20011, "insert task failed, reason: " + message); + } + String taskStr = object.toString(); + Long taskID = Long.parseLong(taskStr); + jobRequest.setId(taskID); + + } else { + //Todo can be throw exception if it is not requestPersistTask(todo 如果不是 requestPersistTask的话，可以进行throw异常) + } + } + + + @Override + public void persist(SubJobInfo subjobInfo) throws ErrorException { + + } + + // @Override +// public Task retrieve(Long taskID)throws EntranceIllegalParamException, QueryFailedException, EntranceRPCException { +// +// Task task = null; +// +// if ( taskID == null || taskID < 0){ +// throw new EntranceIllegalParamException(20003, "taskID can't be null or less than 0"); +// } +// +// RequestQueryTask requestQueryTask = new RequestQueryTask(); +// requestQueryTask.setTaskID(taskID); +// ResponsePersist responsePersist = null; +// try { +//// responsePersist = (ResponsePersist) sender.ask(requestQueryTask); +// responsePersist = workflowQueryService.query(requestQueryTask); +// }catch(Exception e){ +// logger.error("Requesting the corresponding task failed with taskID: {}(通过taskID: {} 请求相应的task失败)", taskID, e); +// throw new EntranceRPCException(20020, "sender rpc failed", e); +// } +// int status = responsePersist.getStatus(); +// //todo I want to discuss it again.(要再讨论下) +// String message = responsePersist.getMsg(); +// if (status != 0){ +// logger.error("By taskID: {} request the corresponding task return status code is not 0, the query fails(通过taskID: {} 请求相应的task返回状态码不为0，查询失败)", taskID); +// throw new QueryFailedException(20010, "retrieve task failed, reason: " + message); +// } +// Map data = responsePersist.getData(); +// if (data != null){ +// Object object = data.get(TaskConstant.TASK); +// if (object instanceof List){ +// List list = (List)object; +// if (list.size() == 0){ +// logger.info("returned list length is 0, maybe there is no task corresponding to {}", taskID); +// }else if (list.size() == 1){ +// Object t = list.get(0); +// Gson gson = new Gson(); +// String json = gson.toJson(t); +// task = gson.fromJson(json, RequestPersistTask.class); +// } +// } +// } +// return task; +// } + + + @Override + public void updateIfNeeded(JobRequest jobRequest) throws EntranceRPCException, EntranceIllegalParamException { + if (jobRequest == null) { + throw new EntranceIllegalParamException(20004, "task can not be null, unable to do update operation"); + } + + RequestUpdateTask requestUpdateTask = new RequestUpdateTask(); + RequestPersistTask requestPersistTask = FlowExecutionUtils.jobRequest2RequestPersistTask(jobRequest); + BeanUtils.copyProperties(requestPersistTask, requestUpdateTask); + try { + workflowQueryService.change(requestUpdateTask); + } catch (Exception e) { + logger.error("Request to update task with taskID {} failed, possibly due to RPC failure(请求更新taskID为 {} 的任务失败，原因可能是RPC失败)", requestUpdateTask.getTaskID(), e); + throw new EntranceRPCException(20020, "sender rpc failed ", e); + } + + } + + @Override + public void updateIfNeeded(SubJobInfo subJobInfo) throws ErrorException { + + } + + @Override + public Task[] readAll(String instance) throws EntranceIllegalParamException, EntranceRPCException, QueryFailedException { + + List retList = new ArrayList<>(); + + if (instance == null || "".equals(instance)) { + throw new EntranceIllegalParamException(20004, "instance can not be null"); + } + + RequestReadAllTask requestReadAllTask = new RequestReadAllTask(instance); + ResponsePersist responsePersist = null; + try { + //todo 目前没有使用到 +// responsePersist = (ResponsePersist)sender.ask(requestReadAllTask); +// workflowQueryService. + } catch (Exception e) { + throw new EntranceRPCException(20020, "sender rpc failed ", e); + } + if (responsePersist != null) { + int status = responsePersist.getStatus(); + String message = responsePersist.getMsg(); + if (status != 0) { + throw new QueryFailedException(20011, "read all tasks failed, reason: " + message); + } + Map data = responsePersist.getData(); + Object object = data.get(TaskConstant.TASK); + if (object instanceof List) { + List list = (List) object; + if (list.size() == 0) { + logger.info("no running task in this instance: {}", instance); + } + for (Object o : list) { + if (o instanceof RequestPersistTask) { + retList.add((RequestPersistTask) o); + } + } + } + } + return retList.toArray(new Task[0]); + } + + @Override + public JobRequest retrieveJobReq(Long jobGroupId) throws ErrorException { + return null; + } + + @Override + public SubJobDetail retrieveJobDetailReq(Long jobDetailId) throws ErrorException { + return null; + } + + @Override + public void close() throws IOException { + + } + + @Override + public void flush() throws IOException { + + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowEntranceRestfulApi.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowEntranceRestfulApi.java new file mode 100644 index 000000000..0b2dab557 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowEntranceRestfulApi.java @@ -0,0 +1,138 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.restful; + +import com.webank.wedatasphere.dss.common.entity.DSSWorkspace; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.linkis.common.log.LogUtils; +import com.webank.wedatasphere.linkis.entrance.EntranceServer; +import com.webank.wedatasphere.linkis.entrance.annotation.EntranceServerBeanAnnotation; +import com.webank.wedatasphere.linkis.entrance.execute.EntranceJob; +import com.webank.wedatasphere.linkis.entrance.restful.EntranceRestfulApi; +import com.webank.wedatasphere.linkis.entrance.utils.JobHistoryHelper; +import com.webank.wedatasphere.linkis.governance.common.entity.job.JobRequest; +import com.webank.wedatasphere.linkis.protocol.constants.TaskConstant; +import com.webank.wedatasphere.linkis.protocol.utils.ZuulEntranceUtils; +import com.webank.wedatasphere.linkis.rpc.Sender; +import com.webank.wedatasphere.linkis.scheduler.queue.Job; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; +import scala.Option; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.*; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.Response; +import java.util.Map; + + +@Path("/dss/flow/entrance") +@Component +public class FlowEntranceRestfulApi extends EntranceRestfulApi { + + private EntranceServer entranceServer; + + private static final Logger logger = LoggerFactory.getLogger(FlowEntranceRestfulApi.class); + + @Override + @EntranceServerBeanAnnotation.EntranceServerAutowiredAnnotation + public void setEntranceServer(EntranceServer entranceServer){ + super.setEntranceServer(entranceServer); + this.entranceServer = entranceServer; + } + + /** + * The execute function handles the request submitted by the user to execute the task, and the execution ID is returned to the user. + * execute函数处理的是用户提交执行任务的请求，返回给用户的是执行ID + * json Incoming key-value pair(传入的键值对) + * Repsonse + */ + @Override + @POST + @Path("/execute") + public Response execute(@Context HttpServletRequest req, Map json) { + Message message = null; +// try{ + logger.info("Begin to get an execID"); + DSSWorkspace workspace = SSOHelper.getWorkspace(req); + json.put(TaskConstant.UMUSER, SecurityFilter.getLoginUsername(req)); + Map params = (Map) json.get("params"); + params.put("workspace", workspace); + String label = ((Map) json.get(DSSCommonUtils.DSS_LABELS_KEY)).get("route").toString(); + params.put(DSSCommonUtils.DSS_LABELS_KEY, label); + String execID = entranceServer.execute(json); + Job job = entranceServer.getJob(execID).get(); + JobRequest task = ((EntranceJob) job).getJobRequest(); + Long taskID = task.getId(); + pushLog(LogUtils.generateInfo("You have submitted a new job, script code (after variable substitution) is"), job); + pushLog("************************************SCRIPT CODE************************************", job); + pushLog(task.getExecutionCode(), job); + pushLog("************************************SCRIPT CODE************************************", job); + pushLog(LogUtils.generateInfo("Your job is accepted, jobID is " + execID + " and taskID is " + taskID + ". Please wait it to be scheduled"), job); + execID = ZuulEntranceUtils.generateExecID(execID, Sender.getThisServiceInstance().getApplicationName(), new String[]{Sender.getThisInstance()}); + message = Message.ok(); + message.setMethod("/api/entrance/execute"); + message.data("execID", execID); + message.data("taskID", taskID); + logger.info("End to get an an execID: {}, taskID: {}", execID, taskID); +// }catch(ErrorException e){ +// message = Message.error(e.getDesc()); +// message.setStatus(1); +// message.setMethod("/api/entrance/execute"); +// } + return Message.messageToResponse(message); + + } + + @Override + @GET + @Path("/{id}/status") + public Response status(@PathParam("id") String id, @QueryParam("taskID") String taskID) { + Message message = null; + String realId = ZuulEntranceUtils.parseExecID(id)[3]; + Option job = Option.apply(null); + try { + job = entranceServer.getJob(realId); + } catch (Exception e) { + logger.warn("获取任务 {} 状态时出现错误", realId, e); + //如果获取错误了,证明在内存中已经没有了,去jobhistory找寻一下taskID代表的任务的状态，然后返回 + long realTaskID = Long.parseLong(taskID); + String status = JobHistoryHelper.getStatusByTaskID(realTaskID); + message = Message.ok(); + message.setMethod("/api/entrance/" + id + "/status"); + message.data("status", status).data("execID", id); + return Message.messageToResponse(message); + } + if (job.isDefined()) { + message = Message.ok(); + message.setMethod("/api/entrance/" + id + "/status"); + message.data("status", job.get().getState().toString()).data("execID", id); + } else { + message = Message.error("ID The corresponding job is empty and cannot obtain the corresponding task status.(ID 对应的job为空，不能获取相应的任务状态)"); + } + return Message.messageToResponse(message); + } + + private void pushLog(String log, Job job) { + entranceServer.getEntranceContext().getOrCreateLogManager().onLogUpdate(job, log); + } + +} diff --git a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowExecutionRestfulApi.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowExecutionRestfulApi.java similarity index 93% rename from dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowExecutionRestfulApi.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowExecutionRestfulApi.java index 8ad6dcb51..fe5182e51 100644 --- a/dss-flow-execution-entrance/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowExecutionRestfulApi.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/restful/FlowExecutionRestfulApi.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,7 +16,6 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.restful; -import com.webank.wedatasphere.dss.flow.execution.entrance.FlowContext; import com.webank.wedatasphere.dss.flow.execution.entrance.FlowContext$; import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob; import com.webank.wedatasphere.linkis.entrance.EntranceServer; @@ -27,7 +25,6 @@ import com.webank.wedatasphere.linkis.server.Message; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.springframework.context.annotation.ComponentScan; import org.springframework.stereotype.Component; import scala.Option; @@ -37,11 +34,8 @@ import java.util.List; import java.util.Map; -/** - * Created by peacewong on 2019/11/5. - */ -@Path("/entrance") +@Path("/dss/flow/entrance") @Component @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.APPLICATION_JSON) diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/service/WorkflowQueryService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/service/WorkflowQueryService.java new file mode 100644 index 000000000..bdbfcb0c9 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/service/WorkflowQueryService.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.service; + +import com.webank.wedatasphere.linkis.governance.common.entity.task.*; + + +public interface WorkflowQueryService { + + ResponsePersist add(RequestInsertTask requestInsertTask); + + ResponsePersist change(RequestUpdateTask requestUpdateTask); + +// ResponsePersist query(RequestQueryTask requestQueryTask); +// +// WorkflowQueryTaskVO getTaskByID(Long taskID, String userName); +// +// List search(Long taskID, String username, String status, Date sDate, Date eDate, String executeApplicationName); +// +// RequestPersistTask searchOne(String instance, String execId, Date sDate, Date eDate, String executeApplicationName); +// +// List getQueryVOList(List list); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/service/impl/WorkflowQueryServiceImpl.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/service/impl/WorkflowQueryServiceImpl.scala new file mode 100644 index 000000000..2cdfbc159 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/service/impl/WorkflowQueryServiceImpl.scala @@ -0,0 +1,236 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.service.impl + +import java.lang.Long +import java.util +import java.util.Date + +import com.google.common.collect.Iterables +import com.webank.wedatasphere.dss.flow.execution.entrance.dao.TaskMapper +import com.webank.wedatasphere.dss.flow.execution.entrance.entity.{WorkflowQueryTask, WorkflowQueryTaskVO} +import com.webank.wedatasphere.dss.flow.execution.entrance.exception.FlowQueryErrorException +import com.webank.wedatasphere.dss.flow.execution.entrance.service.WorkflowQueryService +import com.webank.wedatasphere.dss.flow.execution.entrance.status.TaskStatus +import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import com.webank.wedatasphere.linkis.governance.common.entity.task._ +import com.webank.wedatasphere.linkis.manager.label.utils.LabelUtils +import com.webank.wedatasphere.linkis.protocol.utils.ZuulEntranceUtils +import org.apache.commons.lang3.StringUtils +import org.springframework.beans.BeanUtils +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.stereotype.Service +import org.springframework.transaction.annotation.Transactional + + +@Service +class WorkflowQueryServiceImpl extends WorkflowQueryService with Logging { + @Autowired + private var taskMapper: TaskMapper = _ +// @Autowired +// private var queryCacheService: QueryCacheService = _ + + override def add(requestInsertTask: RequestInsertTask): ResponsePersist = { + info("Insert data into the database(往数据库中插入数据)：" + requestInsertTask.toString) +// QueryUtils.storeExecutionCode(requestInsertTask) + val persist = new ResponsePersist + Utils.tryCatch { + val queryTask = requestPersistTaskTask2QueryTask(requestInsertTask) + taskMapper.insertTask(queryTask) + val map = new util.HashMap[String, Object]() + map.put("taskID", queryTask.getTaskID()) + persist.setStatus(0) + persist.setData(map) + } { + case e: Exception => + error(e.getMessage) + persist.setStatus(1) + persist.setMsg(e.getMessage) + } + persist + } + + @Transactional + override def change(requestUpdateTask: RequestUpdateTask): ResponsePersist = { + val executionCode = requestUpdateTask.getExecutionCode + requestUpdateTask.setExecutionCode(null) + info("Update data to the database(往数据库中更新数据)：" + requestUpdateTask.toString) + /*//TODO cooperyang 去查数据内容 + if (StringUtils.isNotEmpty(requestUpdateTask.getEngineInstance)) { + requestUpdateTask.setTaskResource(taskMapper.getResource(requestUpdateTask.getEngineInstance)); + }*/ + val persist = new ResponsePersist + Utils.tryCatch { + if (requestUpdateTask.getErrDesc != null) { + if (requestUpdateTask.getErrDesc.length > 256) { + info(s"errorDesc is too long,we will cut some message") + requestUpdateTask.setErrDesc(requestUpdateTask.getErrDesc.substring(0, 256)) + info(s"${requestUpdateTask.getErrDesc}") + } + } + if (requestUpdateTask.getStatus != null) { + val oldStatus: String = taskMapper.selectTaskStatusForUpdate(requestUpdateTask.getTaskID) + if (oldStatus != null && !shouldUpdate(oldStatus, requestUpdateTask.getStatus)) + throw new FlowQueryErrorException(100639,s"${requestUpdateTask.getTaskID}数据库中的task状态为：${oldStatus}更新的task状态为：${requestUpdateTask.getStatus}更新失败！") + } + taskMapper.updateTask(requestPersistTaskTask2QueryTask(requestUpdateTask)) + + //updated by shanhuang to write cache + //todo add cache + if (TaskStatus.Succeed.toString.equals(requestUpdateTask.getStatus)) { + info("Write cache for task: " + requestUpdateTask.getTaskID) + requestUpdateTask.setExecutionCode(executionCode) +// queryCacheService.writeCache(requestUpdateTask) + } + + val map = new util.HashMap[String, Object] + map.put("taskID", requestUpdateTask.getTaskID) + persist.setStatus(0) + persist.setData(map) + } { + case e: Exception => + error(e.getMessage) + persist.setStatus(1); + persist.setMsg(e.getMessage); + } + persist + } + + def query(requestQueryTask: RequestQueryTask): ResponsePersist = { + info("查询历史task：" + requestQueryTask.toString) + val persist = new ResponsePersist + Utils.tryCatch { + val task = taskMapper.selectTask(requestPersistTaskTask2QueryTask(requestQueryTask)) + val map = new util.HashMap[String, Object]() + map.put("task", queryTaskList2RequestPersistTaskList(task)) + persist.setStatus(0) + persist.setData(map) + } { + case e: Exception => + error(e.getMessage) + persist.setStatus(1); + persist.setMsg(e.getMessage); + } + persist + } + + private def queryTaskList2RequestPersistTaskList(queryTask: java.util.List[WorkflowQueryTask]): java.util.List[RequestPersistTask] = { + import scala.collection.JavaConversions._ + val tasks = new util.ArrayList[RequestPersistTask] + queryTask.foreach(f => tasks.add(queryTask2RequestPersistTask(f))) + tasks + } + + def queryTask2RequestPersistTask(queryTask: WorkflowQueryTask): RequestPersistTask = { +// QueryUtils.exchangeExecutionCode(queryTask) + val task = new RequestPersistTask + BeanUtils.copyProperties(queryTask, task) + task.setSource(FlowExecutionUtils.gson.fromJson(queryTask.getSourceJson, classOf[java.util.HashMap[String, String]])) + task.setParams(FlowExecutionUtils.gson.fromJson(queryTask.getParamsJson, classOf[java.util.HashMap[String, Object]])) + task + } + + private def requestPersistTaskTask2QueryTask(requestPersistTask: RequestPersistTask): WorkflowQueryTask = { + val task: WorkflowQueryTask = new WorkflowQueryTask + BeanUtils.copyProperties(requestPersistTask, task) + if (requestPersistTask.getSource != null) + task.setSourceJson(FlowExecutionUtils.gson.toJson(requestPersistTask.getSource)) + if (requestPersistTask.getParams != null) + task.setParamsJson(FlowExecutionUtils.gson.toJson(requestPersistTask.getParams)) + if (requestPersistTask.getLabels != null) + task.setLabelJson(FlowExecutionUtils.gson.toJson(LabelUtils.labelsToMap(requestPersistTask.getLabels))) + task + } + +// override def getTaskByID(taskID: Long, userName: String): WorkflowQueryTaskVO = { +// val task = new WorkflowQueryTask +// task.setTaskID(taskID) +// task.setUmUser(userName) +// val taskR = taskMapper.selectTask(task) +// import scala.collection.JavaConversions._ +// if (taskR.isEmpty) null else queryTask2QueryTaskVO(taskR(0)) +// } + + + def queryTask2QueryTaskVO(queryTask: WorkflowQueryTask): WorkflowQueryTaskVO = { +// QueryUtils.exchangeExecutionCode(queryTask) + val taskVO = new WorkflowQueryTaskVO + BeanUtils.copyProperties(queryTask, taskVO) + if (!StringUtils.isEmpty(taskVO.getSourceJson)) { + Utils.tryCatch { + val source = FlowExecutionUtils.gson.fromJson(taskVO.getSourceJson, classOf[util.Map[String, String]]) + import scala.collection.JavaConversions._ + taskVO.setSourceTailor(source.map(_._2).foldLeft("")(_ + _ + "-").stripSuffix("-")) + } { + case _ => warn("sourceJson deserializae failed,this task may be the old data") + } + } + if (queryTask.getExecId() != null && queryTask.getExecuteApplicationName() != null && queryTask.getInstance() != null) { + taskVO.setStrongerExecId(ZuulEntranceUtils.generateExecID(queryTask.getExecId(), + queryTask.getExecuteApplicationName(), queryTask.getInstance(), queryTask.getRequestApplicationName)) + } + val status = queryTask.getStatus() + val createdTime = queryTask.getCreatedTime() + val updatedTime = queryTask.getUpdatedTime() + if (isJobFinished(status) && createdTime != null && updatedTime != null) { + taskVO.setCostTime(queryTask.getUpdatedTime().getTime() - queryTask.getCreatedTime().getTime()); + } else if (createdTime != null) { + taskVO.setCostTime(System.currentTimeMillis() - queryTask.getCreatedTime().getTime()); + } + taskVO + } + + def isJobFinished(status: String): Boolean = { + TaskStatus.Succeed.toString.equals(status) || + TaskStatus.Failed.toString.equals(status) || + TaskStatus.Cancelled.toString.equals(status) || + TaskStatus.Timeout.toString.equals(status) + } + +// override def search(taskID: Long, username: String, status: String, sDate: Date, eDate: Date, executeApplicationName: String): util.List[WorkflowQueryTask] = { +// import scala.collection.JavaConversions._ +// val split: util.List[String] = if (status != null) status.split(",").toList else null +// taskMapper.search(taskID, username, split, sDate, eDate, executeApplicationName, null, null) +// } + + def getQueryVOList(list: java.util.List[WorkflowQueryTask]): java.util.List[WorkflowQueryTaskVO] = { + val ovs = new util.ArrayList[WorkflowQueryTaskVO] + import scala.collection.JavaConversions._ + list.foreach(f => { + ovs.add(queryTask2QueryTaskVO(f)) + }) + ovs + } + + private def shouldUpdate(oldStatus: String, newStatus: String): Boolean = TaskStatus.valueOf(oldStatus).ordinal <= TaskStatus.valueOf(newStatus).ordinal + +// override def searchOne(instance: String, execId: String, sDate: Date, eDate: Date, executeApplicationName: String): RequestPersistTask = { +// Iterables.getFirst( +// queryTaskList2RequestPersistTaskList(taskMapper.search(null, null, null, sDate, eDate, executeApplicationName, instance, execId)), +// { +// val requestPersistTask = new RequestPersistTask +// requestPersistTask.setInstance(instance) +// requestPersistTask.setExecId(execId) +// requestPersistTask.setStatus(TaskStatus.Inited.toString) +// requestPersistTask.setProgress(0.0f) +// requestPersistTask.setTaskID(0l) +// requestPersistTask +// }) +// } +} + diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/status/TaskStatus.java b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/status/TaskStatus.java new file mode 100644 index 000000000..f783999ee --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/java/com/webank/wedatasphere/dss/flow/execution/entrance/status/TaskStatus.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.status; + + +public enum TaskStatus { + Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/DSSFowExecutionServerApplication.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/DSSFowExecutionServerApplication.scala new file mode 100644 index 000000000..732fe6e34 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/DSSFowExecutionServerApplication.scala @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance + +import com.webank.wedatasphere.dss.common.utils.DSSMainHelper +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} + + +object DSSFowExecutionServerApplication extends Logging { + + val userName: String = System.getProperty("user.name") + val hostName: String = Utils.getComputerName + + def main(args: Array[String]): Unit = { + val serviceName = System.getProperty("serviceName")//ProjectConf.SERVICE_NAME.getValue + DSSMainHelper.formatPropertyFiles(serviceName) + val allArgs = args ++ DSSMainHelper.getExtraSpringOptions + System.setProperty("hostName", hostName) + System.setProperty("userName", userName) + info(s"Ready to start $serviceName with args: ${allArgs.toList}.") + println(s"Test Ready to start $serviceName with args: ${allArgs.toList}.") + DataWorkCloudApplication.main(allArgs) + } +} \ No newline at end of file diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContext.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContext.scala similarity index 90% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContext.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContext.scala index f03081a5e..f791be591 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContext.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContext.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,13 +18,11 @@ package com.webank.wedatasphere.dss.flow.execution.entrance import java.util -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode import com.webank.wedatasphere.dss.flow.execution.entrance.node.NodeRunner +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode import com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEventState.SchedulerEventState -/** - * Created by peacewong on 2019/11/5. - */ + trait FlowContext { def getRunningNodes: util.Map[String, NodeRunner] @@ -52,8 +49,8 @@ object FlowContext { } def changedNodeState(fromMap: util.Map[String, NodeRunner], - toMap: util.Map[String, NodeRunner], node: SchedulerNode,info:String): Unit = { - val nodeName = node.getDssNode.getName + toMap: util.Map[String, NodeRunner], node: WorkflowNode, info:String): Unit = { + val nodeName = node.getDSSNode.getName if (fromMap.containsKey(nodeName)) { val runner = fromMap.get(nodeName) runner.setNodeExecutedInfo(info) @@ -78,6 +75,7 @@ object FlowContext { nodeView.put("nodeID", node.getId) nodeView.put("info", nodeRunner.getNodeExecutedInfo()) nodeView.put("startTime", nodeRunner.getStartTime()) + nodeView.put("nowTime", nodeRunner.getNowTime()) nodes.add(nodeView) } } diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContextImpl.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContextImpl.scala similarity index 94% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContextImpl.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContextImpl.scala index 84f6d7caa..a1344e3da 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContextImpl.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/FlowContextImpl.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -24,10 +23,7 @@ import com.webank.wedatasphere.linkis.common.utils.Logging import com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEventState import com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEventState.SchedulerEventState -/** - * created by chaogefeng on 2019/11/5 15:59 - * Description: - */ + class FlowContextImpl extends FlowContext with Logging { val runningNodes: util.Map[String, NodeRunner] = new util.HashMap[String, NodeRunner] diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceConfiguration.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceConfiguration.scala new file mode 100644 index 000000000..27a2cce36 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceConfiguration.scala @@ -0,0 +1,76 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.conf + +import com.webank.wedatasphere.linkis.common.conf.CommonVars + + +object FlowExecutionEntranceConfiguration { + + val WORKFLOW_APPLICATION_NAME = CommonVars("wds.dss.workflow.service.name", "DSS-WORKFLOW-SERVER-") + + val NODE_PRINT_FLAG = CommonVars("wds.linkis.flow.node.log.flag", false) + + val NODE_STATUS_POLLER_THREAD_SIZE = CommonVars("wds.dds.flow.node.status.poller.thread.size", 20) + + val NODE_STATUS_POLLER_SCHEDULER_TIME = CommonVars("wds.dds.flow.node.status.poller.scheduler.time", 5) + val FLOW_EXECUTION_SCHEDULER_POOL_SIZE = CommonVars("wds.linkis.flow.execution.pool.size", 30) + + val NODE_STATUS_INTERVAL = CommonVars("wds.dds.flow.node.status.poller.interval.time", 3000) + + val COMMAND = "command" + + val JOB_ID = "job.id" + + val FLOW_NAME = "dss.flow.flowid" + + + val PROJECT_NAME = "dss.flow.projectname" + + val FLOW_EXEC_ID = "dss.flow.execid" + + val WORKSPACE_ID = "dss.flow.workspace.id" + + + val PROXY_USER = "user.to.proxy" + + + val FLOW_SUBMIT_USER = "flow.submituser" + + + val SKIP_NODES = CommonVars("wds.dss.flow.skip.nodes", "workflow.subflow") + + val SIGNAL_NODES = CommonVars("wds.dss.flow.signal.nodes", "linkis.appconn.eventchecker.eventreceiver") + + val PROPS_MAP = "props_map" + + val WORKSPACE = "workspace" + + val CONTEXT_ID = "contextID" + + val NODE_NAME = "nodeName" + + val FLOW_VAR_MAP = "flow_var_map" + + + val NODE_CONFIGURATION_KEY = "configuration" + + val PROJECT_RESOURCES = "project_resources" + + val FLOW_RESOURCES = "flow_resources" + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceSpringConfiguration.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceSpringConfiguration.scala new file mode 100644 index 000000000..36dceee15 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/conf/FlowExecutionEntranceSpringConfiguration.scala @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.conf + +import com.webank.wedatasphere.dss.flow.execution.entrance.engine.{FlowEntranceEngine, FlowExecutionExecutorManagerImpl} +import com.webank.wedatasphere.dss.flow.execution.entrance.entranceparser.FlowExecutionParser +import com.webank.wedatasphere.dss.flow.execution.entrance.persistence.WorkflowPersistenceEngine +import com.webank.wedatasphere.linkis.entrance.EntranceParser +import com.webank.wedatasphere.linkis.entrance.annotation.{EntranceExecutorManagerBeanAnnotation, EntranceParserBeanAnnotation, PersistenceEngineBeanAnnotation, PersistenceManagerBeanAnnotation} +import com.webank.wedatasphere.linkis.entrance.persistence.{PersistenceEngine, PersistenceManager} +import com.webank.wedatasphere.linkis.scheduler.executer.ExecutorManager +import org.slf4j.LoggerFactory +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.context.annotation.Configuration + + +@Configuration +class FlowExecutionEntranceSpringConfiguration { + private val logger = LoggerFactory.getLogger(classOf[FlowExecutionEntranceSpringConfiguration]) + + @EntranceExecutorManagerBeanAnnotation + def generateExternalEntranceExecutorManager(@Autowired flowEntranceEngine:FlowEntranceEngine): ExecutorManager = { + logger.info("begin to get FlowExecution Entrance EntranceExecutorManager") + new FlowExecutionExecutorManagerImpl(flowEntranceEngine) + } + + @PersistenceEngineBeanAnnotation + def generatePersistenceEngine(): PersistenceEngine ={ + new WorkflowPersistenceEngine() + } + + @EntranceParserBeanAnnotation + def generateEntranceParser(@PersistenceManagerBeanAnnotation.PersistenceManagerAutowiredAnnotation persistenceManager: PersistenceManager): EntranceParser = { + logger.info("begin to get FlowExecution Entrance parser") + new FlowExecutionParser(persistenceManager) + } + + +} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowEntranceEngine.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowEntranceEngine.scala similarity index 77% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowEntranceEngine.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowEntranceEngine.scala index 38042fc01..338a33bb2 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowEntranceEngine.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowEntranceEngine.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -21,17 +20,12 @@ import java.util import com.webank.wedatasphere.dss.flow.execution.entrance.exception.FlowExecutionErrorException import com.webank.wedatasphere.dss.flow.execution.entrance.execution.FlowExecution -import com.webank.wedatasphere.dss.flow.execution.entrance.job.{FlowEntranceJob, FlowExecutionRequest} import com.webank.wedatasphere.dss.flow.execution.entrance.job.parser.FlowEntranceJobParser +import com.webank.wedatasphere.dss.flow.execution.entrance.job.{FlowEntranceJob, FlowExecutionRequest} import com.webank.wedatasphere.dss.flow.execution.entrance.resolver.FlowDependencyResolver import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import com.webank.wedatasphere.linkis.entrance.EntranceServer -import com.webank.wedatasphere.linkis.entrance.annotation.EntranceServerBeanAnnotation -import com.webank.wedatasphere.linkis.entrance.execute.{EngineExecuteAsynReturn, EntranceEngine, StorePathExecuteRequest} -import com.webank.wedatasphere.linkis.entrance.persistence.EntranceResultSetEngine -import com.webank.wedatasphere.linkis.protocol.engine.{JobProgressInfo, RequestTask} +import com.webank.wedatasphere.linkis.scheduler.executer.ExecutorState.ExecutorState import com.webank.wedatasphere.linkis.scheduler.executer._ -import org.apache.commons.lang.StringUtils import org.springframework.beans.factory.annotation.Autowired import org.springframework.stereotype.Component @@ -39,8 +33,9 @@ import scala.collection.JavaConversions._ @Component -class FlowEntranceEngine extends EntranceEngine(id = 0) with ConcurrentTaskOperateSupport with Logging { +class FlowEntranceEngine extends Executor with ConcurrentTaskOperateSupport with Logging { + private var id: Long = _ @Autowired private var flowParsers: util.List[FlowEntranceJobParser] = _ @@ -51,6 +46,7 @@ class FlowEntranceEngine extends EntranceEngine(id = 0) with ConcurrentTaskOpera @Autowired private var flowExecution: FlowExecution = _ + private val SUBMIT_JOB_LOCK = "STATUS_CHANGED_LOCK".intern() protected def executeLine(code: String, storePath: String, alias: String): ExecuteResponse = { val realCode = code.trim() @@ -78,10 +74,12 @@ class FlowEntranceEngine extends EntranceEngine(id = 0) with ConcurrentTaskOpera throw new FlowExecutionErrorException(90101, s"Failed to parser flow of job(${job.getId})", t) } } - for (flowDependencyResolver <- flowDependencyResolvers) { - flowDependencyResolver.resolvedFlow(job) + this.SUBMIT_JOB_LOCK.synchronized { + for (flowDependencyResolver <- flowDependencyResolvers) { + flowDependencyResolver.resolvedFlow(job) + } + flowExecution.runJob(job) } - flowExecution.runJob(job) } { t => job.kill() error(s"Failed to execute job: ${job.getId}", t) @@ -91,8 +89,6 @@ class FlowEntranceEngine extends EntranceEngine(id = 0) with ConcurrentTaskOpera SuccessExecuteResponse() } - override protected def callExecute(request: RequestTask): EngineExecuteAsynReturn = null - override def kill(jobId: String): Boolean = { true } @@ -120,4 +116,12 @@ class FlowEntranceEngine extends EntranceEngine(id = 0) with ConcurrentTaskOpera override def close(): Unit = { } + + override def getId: Long = this.id + + override def state: ExecutorState = ExecutorState.Idle + + override def getExecutorInfo: ExecutorInfo = { + null + } } diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowExecutionExecutorManagerImpl.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowExecutionExecutorManagerImpl.scala new file mode 100644 index 000000000..32ee283d9 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/engine/FlowExecutionExecutorManagerImpl.scala @@ -0,0 +1,78 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.engine + +import java.util + +import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob +import com.webank.wedatasphere.linkis.common.utils.Logging +import com.webank.wedatasphere.linkis.scheduler.executer.{Executor, ExecutorManager} +import com.webank.wedatasphere.linkis.scheduler.listener.ExecutorListener +import com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEvent + +import scala.concurrent.duration.Duration + + +class FlowExecutionExecutorManagerImpl(flowEntranceEngine: FlowEntranceEngine) extends ExecutorManager with Logging{ + logger.info("FlowExecutionExecutorManagerImpl Registered") + private val idToEngines = new util.HashMap[Long, Executor] + + override def setExecutorListener(executorListener: ExecutorListener): Unit = { + + } + + + + protected override def createExecutor(schedulerEvent: SchedulerEvent): Executor = schedulerEvent match { + case job: FlowEntranceJob => + idToEngines.put(this.flowEntranceEngine.getId, this.flowEntranceEngine) + this.flowEntranceEngine + case _ => null + } + + override def askExecutor(schedulerEvent: SchedulerEvent): Option[Executor] = schedulerEvent match { + case job: FlowEntranceJob => + Some(createExecutor(schedulerEvent)) + } + + override def askExecutor(schedulerEvent: SchedulerEvent, wait: Duration): Option[Executor] = schedulerEvent match { + case job: FlowEntranceJob => + Some(createExecutor(schedulerEvent)) + } + + + override def getById(id: Long): Option[Executor] = { + Option(idToEngines.get(id)) + } + + override def getByGroup(groupName: String): Array[Executor] = { + null + } + + + + override def shutdown(): Unit = {} + + override def delete(executor: Executor): Unit = { + if (null != executor) { + executor.close() + idToEngines.remove(executor.getId) + }else { + logger.info("remove executor failed!") + } + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/entranceparser/FlowExecutionParser.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/entranceparser/FlowExecutionParser.scala new file mode 100644 index 000000000..e5d107b1f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/entranceparser/FlowExecutionParser.scala @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.entranceparser + + +import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob +import com.webank.wedatasphere.linkis.common.utils.Logging +import com.webank.wedatasphere.linkis.entrance.parser.CommonEntranceParser +import com.webank.wedatasphere.linkis.entrance.persistence.PersistenceManager +import com.webank.wedatasphere.linkis.governance.common.entity.job.JobRequest +import com.webank.wedatasphere.linkis.governance.common.entity.task.RequestPersistTask +import com.webank.wedatasphere.linkis.protocol.task.Task +import com.webank.wedatasphere.linkis.scheduler.queue.Job +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ + + +class FlowExecutionParser(persistenceManager: PersistenceManager) extends CommonEntranceParser(persistenceManager) with Logging { + + override def parseToJob(jobRequest: JobRequest): Job = { + val job = new FlowEntranceJob(persistenceManager) + job.setJobRequest(jobRequest) + job.setUser(jobRequest.getExecuteUser) + job.setCreator("FlowEntrance") + job.setParams(jobRequest.getParams.toMap[String, Any]) + job.setEntranceListenerBus(getEntranceContext.getOrCreateEventListenerBus) + job.setListenerEventBus(null) + job.setProgress(0f) + job.setEntranceContext(getEntranceContext) + job + } +} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowExecutionErrorException.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowExecutionErrorException.scala similarity index 87% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowExecutionErrorException.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowExecutionErrorException.scala index 0ad268be4..f61df8d48 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowExecutionErrorException.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowExecutionErrorException.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,9 +18,7 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.exception import com.webank.wedatasphere.linkis.common.exception.ErrorException -/** - * Created by peacewong on 2019/11/5. - */ + class FlowExecutionErrorException(errorCode: Int, desc: String) extends ErrorException(errorCode, desc){ def this(errorCode: Int, desc: String, t:Throwable){ diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowQueryErrorException.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowQueryErrorException.scala new file mode 100644 index 000000000..fbc611f1e --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/exception/FlowQueryErrorException.scala @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.exception + +import com.webank.wedatasphere.linkis.common.exception.ErrorException + + +class FlowQueryErrorException(errorCode: Int, desc: String) extends ErrorException(errorCode, desc){ + + def this(errorCode: Int, desc: String, t:Throwable){ + this(errorCode, desc) + initCause(t) + } + +} \ No newline at end of file diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/DefaultFlowExecution.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/DefaultFlowExecution.scala similarity index 79% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/DefaultFlowExecution.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/DefaultFlowExecution.scala index c1164c9d3..ce8973868 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/DefaultFlowExecution.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/DefaultFlowExecution.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -23,16 +22,13 @@ import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEnt import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob import com.webank.wedatasphere.dss.flow.execution.entrance.node.{NodeExecutionState, NodeRunner} import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import com.webank.wedatasphere.linkis.common.utils.Logging import org.springframework.stereotype.Service import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer -/** - * Created by johnnwang on 2019/11/5. - */ @Service class DefaultFlowExecution extends FlowExecution with Logging { @@ -60,13 +56,20 @@ class DefaultFlowExecution extends FlowExecution with Logging { runner.fromScheduledTunToState(NodeExecutionState.Running) // submit node runner runningNodes.add(runner) - } else { - info(s"This node ${runner.getNode.getDssNode.getName} Skipped in execution") - runner.fromScheduledTunToState(NodeExecutionState.Skipped) + } + } + //避免skip节点导致任务重复提交，所以先要把所有running节点收集完成，改变状态，再来收集skipped节点，skipped节点被多次提交不受影响。 + runners.foreach{ + runner =>{ + if (FlowExecutionUtils.isSkippedNode(runner.getNode)){ + info(s"This node ${runner.getNode.getDSSNode.getName} Skipped in execution") + runner.fromScheduledTunToState(NodeExecutionState.Skipped) + } } } info(s"${flowEntranceJob.getId} Submit nodes(${runningNodes.size}) to running") runningNodes.foreach{ node => + node.getNode.getDSSNode.getParams.get(FlowExecutionEntranceConfiguration.PROPS_MAP).asInstanceOf[java.util.Map[String, Any]].putAll(flowEntranceJob.getParams) node.run() nodeRunnerQueue.put(node) if (pollerCount < FlowExecutionEntranceConfiguration.NODE_STATUS_POLLER_THREAD_SIZE.getValue){ diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/FlowExecution.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/FlowExecution.scala similarity index 86% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/FlowExecution.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/FlowExecution.scala index 2160a01ee..1779dd71e 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/FlowExecution.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/FlowExecution.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,9 +18,7 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.execution import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob -/** - * Created by peacewong on 2019/11/5. - */ + trait FlowExecution { def runJob(flowEntranceJob: FlowEntranceJob) diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/NodeExecutionStatusPoller.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/NodeExecutionStatusPoller.scala similarity index 90% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/NodeExecutionStatusPoller.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/NodeExecutionStatusPoller.scala index ecaa1bf88..da32f0d65 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/NodeExecutionStatusPoller.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/execution/NodeExecutionStatusPoller.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -23,9 +22,7 @@ import java.util.concurrent.LinkedBlockingQueue import com.webank.wedatasphere.dss.flow.execution.entrance.node.NodeRunner import com.webank.wedatasphere.linkis.common.utils.Logging -/** - * Created by peacewong on 2019/11/17. - */ + class NodeExecutionStatusPoller(nodeRunnerQueue: LinkedBlockingQueue[NodeRunner] ) extends Runnable with Logging{ diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowEntranceJob.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowEntranceJob.scala similarity index 86% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowEntranceJob.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowEntranceJob.scala index 42bf395d4..0d1ae58a7 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowEntranceJob.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowEntranceJob.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,52 +16,50 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.job -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.{SchedulerFlow, SchedulerNode} -import com.webank.wedatasphere.dss.common.entity.project.DSSProject import com.webank.wedatasphere.dss.flow.execution.entrance.exception.FlowExecutionErrorException -import com.webank.wedatasphere.dss.flow.execution.entrance.{FlowContext, FlowContextImpl} import com.webank.wedatasphere.dss.flow.execution.entrance.listener.NodeRunnerListener -import com.webank.wedatasphere.dss.flow.execution.entrance.node.{NodeExecutionState, NodeRunner} import com.webank.wedatasphere.dss.flow.execution.entrance.node.NodeExecutionState.NodeExecutionState +import com.webank.wedatasphere.dss.flow.execution.entrance.node.{NodeExecutionState, NodeRunner} +import com.webank.wedatasphere.dss.flow.execution.entrance.{FlowContext, FlowContextImpl} +import com.webank.wedatasphere.dss.workflow.core.entity.{Workflow, WorkflowNode} import com.webank.wedatasphere.linkis.common.log.LogUtils import com.webank.wedatasphere.linkis.common.utils.Utils import com.webank.wedatasphere.linkis.entrance.execute.StorePathExecuteRequest import com.webank.wedatasphere.linkis.entrance.job.EntranceExecutionJob -import com.webank.wedatasphere.linkis.protocol.query.RequestPersistTask +import com.webank.wedatasphere.linkis.entrance.persistence.PersistenceManager import com.webank.wedatasphere.linkis.scheduler.executer.{ErrorExecuteResponse, ExecuteRequest, SuccessExecuteResponse} -import com.webank.wedatasphere.linkis.scheduler.queue.{Job, SchedulerEventState} import com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEventState.Running +import com.webank.wedatasphere.linkis.scheduler.queue.{Job, SchedulerEventState} -import scala.beans.BeanProperty import scala.collection.JavaConversions._ import scala.collection.mutable.ArrayBuffer -class FlowEntranceJob extends EntranceExecutionJob with NodeRunnerListener { +class FlowEntranceJob(persistManager:PersistenceManager) extends EntranceExecutionJob(persistManager) with NodeRunnerListener { - private var flow: SchedulerFlow = _ + private var flow: Workflow = _ private val flowContext: FlowContext = new FlowContextImpl - @BeanProperty var dwsProject: DSSProject = _ +// @BeanProperty var dwsProject: DWSProject = _ - def setFlow(flow: SchedulerFlow): Unit = this.flow = flow + def setFlow(flow: Workflow): Unit = this.flow = flow - def getFlow: SchedulerFlow = this.flow + def getFlow: Workflow = this.flow def getFlowContext: FlowContext = this.flowContext private val STATUS_CHANGED_LOCK = "STATUS_CHANGED_LOCK".intern() + override def init():Unit = { + + } override def jobToExecuteRequest(): ExecuteRequest = { new ExecuteRequest with StorePathExecuteRequest with FlowExecutionRequest { - override val code: String = FlowEntranceJob.this.getTask match { - case requestPersistTask: RequestPersistTask => requestPersistTask.getExecutionCode - case _ => null - } - override val storePath: String = FlowEntranceJob.this.getTask match { - case requestPersistTask: RequestPersistTask => requestPersistTask.getResultLocation + override val code: String = FlowEntranceJob.this.getJobRequest.getExecutionCode + override val storePath: String = FlowEntranceJob.this.getJobRequest match { +// case requestPersistTask: RequestPersistTask => requestPersistTask.getResultLocation case _ => "" } override val job: Job = FlowEntranceJob.this @@ -78,9 +75,9 @@ class FlowEntranceJob extends EntranceExecutionJob with NodeRunnerListener { } - override def onStatusChanged(fromState: NodeExecutionState, toState: NodeExecutionState, node: SchedulerNode): Unit = { + override def onStatusChanged(fromState: NodeExecutionState, toState: NodeExecutionState, node: WorkflowNode): Unit = { - val nodeName = node.getDssNode.getName + val nodeName = node.getDSSNode.getName toState match { case NodeExecutionState.Failed => printLog(s"Failed to execute node($nodeName),prepare to kill flow job", "ERROR") @@ -138,8 +135,8 @@ class FlowEntranceJob extends EntranceExecutionJob with NodeRunnerListener { override def kill(): Unit = if (! SchedulerEventState.isCompleted(this.getState)) this synchronized { if(! SchedulerEventState.isCompleted(this.getState)){ - Utils.tryAndWarn(this.killNodes) super.kill() + Utils.tryAndWarn(this.killNodes) transitionCompleted(ErrorExecuteResponse(s"execute job(${getId}) failed!", new FlowExecutionErrorException(90101, s"This Flow killed by user") )) } } @@ -171,6 +168,7 @@ class FlowEntranceJob extends EntranceExecutionJob with NodeRunnerListener { node.cancel() } + } diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionRequest.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionRequest.scala similarity index 83% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionRequest.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionRequest.scala index b76afe5b4..b1915e04d 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionRequest.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/FlowExecutionRequest.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,10 +18,7 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.job import com.webank.wedatasphere.linkis.scheduler.queue.Job -/** - * created by chaogefeng on 2019/11/4 16:59 - * Description: - */ + trait FlowExecutionRequest { val job:Job } diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowEntranceJobParser.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowEntranceJobParser.scala similarity index 86% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowEntranceJobParser.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowEntranceJobParser.scala index 4de0b1bf3..8f05dbe35 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowEntranceJobParser.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowEntranceJobParser.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,9 +18,7 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.job.parser import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob -/** - * Created by peacewong on 2019/11/5. - */ + trait FlowEntranceJobParser { def parse(flowEntranceJob: FlowEntranceJob):Unit diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobFlowParser.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobFlowParser.scala new file mode 100644 index 000000000..cfd0b8c89 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobFlowParser.scala @@ -0,0 +1,62 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.job.parser +import com.webank.wedatasphere.dss.common.protocol.RequestQueryWorkFlow +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration +import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob +import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils +import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils +import com.webank.wedatasphere.dss.workflow.common.protocol.ResponseQueryWorkflow +import com.webank.wedatasphere.dss.workflow.core.WorkflowFactory +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow +import com.webank.wedatasphere.linkis.common.utils.Logging +import com.webank.wedatasphere.linkis.rpc.Sender +import org.apache.commons.lang3.StringUtils +import org.springframework.core.annotation.Order +import org.springframework.stereotype.Component + + +@Order(1) +@Component +class FlowJobFlowParser extends FlowEntranceJobParser with Logging { + + override def parse(flowEntranceJob: FlowEntranceJob): Unit = { + info(s"Start to parse flow of Job(${flowEntranceJob.getId}).") + val code = flowEntranceJob.jobToExecuteRequest().code + val label = flowEntranceJob.getParams.getOrDefault(DSSCommonUtils.DSS_LABELS_KEY, DSSCommonUtils.ENV_LABEL_VALUE_DEV).asInstanceOf[String] + val codeLanguageLabel = FlowExecutionUtils.getRunTypeLabel(flowEntranceJob.getJobRequest().getLabels) + if("json" == codeLanguageLabel.getCodeType // 兼容老版本 + || "flow" == codeLanguageLabel.getCodeType) { + val flowExecutionCode =DSSCommonUtils.COMMON_GSON.fromJson(code, classOf[java.util.Map[String, Any]]) + val flowId = flowExecutionCode.get("flowId").asInstanceOf[Number].longValue() + val workflow = getDSSScheduleFlowById(flowEntranceJob.getUser, flowId, label) + flowEntranceJob.setFlow(workflow) + info(s"Finished to parse flow of Job(${flowEntranceJob.getId}).") + } + } + + private def getDSSScheduleFlowById(userName: String , flowId:Long, label:String): Workflow = { + val req = RequestQueryWorkFlow(userName, flowId) + val sendWorkflowName = FlowExecutionEntranceConfiguration.WORKFLOW_APPLICATION_NAME.getValue + label.toUpperCase + logger.info("Send query workflow json to "+ sendWorkflowName) + val response: ResponseQueryWorkflow= Sender.getSender(sendWorkflowName) + .ask(req).asInstanceOf[ResponseQueryWorkflow] + WorkflowFactory.INSTANCE.getJsonToFlowParser.parse(response.getDssFlow) + } + +} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobNodeParser.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobNodeParser.scala similarity index 76% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobNodeParser.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobNodeParser.scala index 2f18486fa..1a3e90af9 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobNodeParser.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJobNodeParser.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -28,46 +27,45 @@ import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUt import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowWithContextImpl import com.webank.wedatasphere.linkis.common.utils.Logging - -import scala.collection.JavaConversions._ +import com.webank.wedatasphere.linkis.entrance.execute.EntranceJob import org.springframework.core.annotation.Order import org.springframework.stereotype.Component -/** - * Created by johnnwang on 2019/11/6. - */ +import scala.collection.JavaConversions._ + -@Order(2) +@Order(10) @Component class FlowJobNodeParser extends FlowEntranceJobParser with Logging{ override def parse(flowEntranceJob: FlowEntranceJob): Unit = { - info(s"${flowEntranceJob.getId} Start to parse node of flow") - val project = flowEntranceJob.getDwsProject + info(s"${flowEntranceJob.getId} Start to parse node of flow.") val flow = flowEntranceJob.getFlow val flowContext = flowEntranceJob.getFlowContext if(null == flow) throw new FlowExecutionErrorException(90101, "This fow of job is empty ") - val nodes = flow.getSchedulerNodes + val nodes = flow.getWorkflowNodes for (node <- nodes) { val nodeName = node.getName val propsMap = new util.HashMap[String, String]() - val proxyUser = if (node.getDssNode.getUserProxy == null) flowEntranceJob.getUser else node.getDssNode.getUserProxy - propsMap.put(PROJECT_NAME, project.getName) + val proxyUser = if (node.getDSSNode.getUserProxy == null) flowEntranceJob.getUser else node.getDSSNode.getUserProxy propsMap.put(FLOW_NAME, flow.getName) propsMap.put(JOB_ID, nodeName) + val projectName =flowEntranceJob.asInstanceOf[EntranceJob].getJobRequest.getSource.get("projectName") + propsMap.put(PROJECT_NAME,projectName) propsMap.put(FLOW_SUBMIT_USER, flowEntranceJob.getUser) - info(s"${flowEntranceJob.getId} nodeName:${nodeName} node:${node.getNodeType}") + info(s"Job(${flowEntranceJob.getId}) with nodeName: $nodeName and node: ${node.getNodeType}.") propsMap.put(LinkisJobExecutionConfiguration.LINKIS_TYPE, node.getNodeType) propsMap.put(PROXY_USER, proxyUser) - propsMap.put(COMMAND, LinkisJobExecutionUtils.gson.toJson(node.getDssNode.getJobContent)) + propsMap.put(COMMAND, LinkisJobExecutionUtils.gson1.toJson(node.getDSSNode.getJobContent)) - var params = node.getDssNode.getParams + var params = node.getDSSNode.getParams if (params == null) { params = new util.HashMap[String,AnyRef]() - node.getDssNode.setParams(params) + node.getDSSNode.setParams(params) } val flowVar = new util.HashMap[String, AnyRef]() val properties = flow.getFlowProperties @@ -79,9 +77,14 @@ class FlowJobNodeParser extends FlowEntranceJobParser with Logging{ propsMap.put(FlowExecutionEntranceConfiguration.FLOW_EXEC_ID, flowEntranceJob.getId) + flow match { + case workflow: WorkflowWithContextImpl => + propsMap.put(CONTEXT_ID, workflow.getContextID) + case _ => + } + params.put(PROPS_MAP, propsMap) params.put(FLOW_VAR_MAP, flowVar) - params.put(PROJECT_RESOURCES, project.getProjectResources) val flowNameAndResources = new util.HashMap[String, util.List[BMLResource]]() flowNameAndResources.put("flow." + flow.getName + LinkisJobExecutionConfiguration.RESOURCES_NAME, FlowExecutionUtils.resourcesAdaptation(flow.getFlowResources)) params.put(FLOW_RESOURCES, flowNameAndResources) diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJsonFlowParser.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJsonFlowParser.scala new file mode 100644 index 000000000..cfe84be72 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/job/parser/FlowJsonFlowParser.scala @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.job.parser +import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob +import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow +import com.webank.wedatasphere.dss.workflow.core.WorkflowFactory +import com.webank.wedatasphere.linkis.common.utils.Logging +import com.webank.wedatasphere.linkis.governance.common.entity.job.JobRequest +import org.springframework.core.annotation.Order + +import scala.collection.convert.wrapAsScala._ + + +@Order(2) +class FlowJsonFlowParser extends FlowEntranceJobParser with Logging { + + override def parse(flowEntranceJob: FlowEntranceJob): Unit = { + info(s"Start to parse flow of Job(${flowEntranceJob.getId}).") + val codeLanguageLabel = FlowExecutionUtils.getRunTypeLabel(flowEntranceJob.getJobRequest().getLabels) + if("flowjson" == codeLanguageLabel.getCodeType.toLowerCase) { + val workflow = WorkflowFactory.INSTANCE.getJsonToFlowParser.parse(createDSSFlow(flowEntranceJob.getJobRequest())) + flowEntranceJob.setFlow(workflow) + info(s"Finished to parse flow of Job(${flowEntranceJob.getId}).") + } + } + + private def createDSSFlow(jobRequest: JobRequest): DSSFlow = { + val dssFlow = new DSSFlow + dssFlow.setFlowJson(jobRequest.getExecutionCode) + dssFlow.setCreator(jobRequest.getExecuteUser) + val name = if(jobRequest.getSource != null) jobRequest.getSource.values().mkString("-") + else jobRequest.getSubmitUser + "-Flow" + dssFlow.setName(name) + dssFlow + } + +} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/listener/NodeRunnerListener.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/listener/NodeRunnerListener.scala similarity index 76% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/listener/NodeRunnerListener.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/listener/NodeRunnerListener.scala index 4d25360f3..760cf3740 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/listener/NodeRunnerListener.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/listener/NodeRunnerListener.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,13 +16,11 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.listener -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode import com.webank.wedatasphere.dss.flow.execution.entrance.node.NodeExecutionState.NodeExecutionState +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode + -/** - * Created by peacewong on 2019/11/5. - */ trait NodeRunnerListener { - def onStatusChanged(fromState: NodeExecutionState, toState: NodeExecutionState, node: SchedulerNode): Unit + def onStatusChanged(fromState: NodeExecutionState, toState: NodeExecutionState, node: WorkflowNode): Unit } diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/log/FlowExecutionLog.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/log/FlowExecutionLog.scala similarity index 92% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/log/FlowExecutionLog.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/log/FlowExecutionLog.scala index 8a317bbae..a26e2b8f1 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/log/FlowExecutionLog.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/log/FlowExecutionLog.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -22,9 +21,7 @@ import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEnt import com.webank.wedatasphere.dss.linkis.node.execution.log.LinkisJobExecutionLog import com.webank.wedatasphere.linkis.common.utils.Logging -/** - * Created by peacewong on 2019/11/5. - */ + class FlowExecutionLog(log:Logging) extends LinkisJobExecutionLog { override def info(message: scala.Any, t: Throwable): Unit = { if(message != null && FlowExecutionEntranceConfiguration.NODE_PRINT_FLAG.getValue) log.info(message.toString, t) diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/AppConnJobBuilder.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/AppConnJobBuilder.scala new file mode 100644 index 000000000..992750606 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/AppConnJobBuilder.scala @@ -0,0 +1,121 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.node + +import java.util + +import com.webank.wedatasphere.dss.common.entity.Resource +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration +import com.webank.wedatasphere.dss.flow.execution.entrance.job._ +import com.webank.wedatasphere.dss.flow.execution.entrance.utils.FlowExecutionUtils +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration +import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource +import com.webank.wedatasphere.dss.linkis.node.execution.job._ +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode +import org.apache.commons.lang.StringUtils + + +object AppConnJobBuilder { + + def builder():FlowBuilder = new FlowBuilder + + class FlowBuilder extends Builder { + + private var jobProps: java.util.Map[String, String] = _ + + private var node: WorkflowNode = _ + + def setJobProps(jobProps: java.util.Map[String, String]): FlowBuilder = { + this.jobProps = jobProps + this + } + + def setNode(node: WorkflowNode): FlowBuilder = { + this.node = node + this + } + + override protected def getJobType: String = jobProps.getOrDefault(LinkisJobExecutionConfiguration.LINKIS_TYPE, LinkisJobExecutionConfiguration.LINKIS_DEFAULT_TYPE.getValue(jobProps)) + + override protected def creatLinkisJob(isLinkisType: Boolean): LinkisJob = { + if(isLinkisType){ + val linkisJob = new FlowExecutionCommonLinkisJob + linkisJob.setJobProps(this.jobProps) + linkisJob + } else { + val linkisJob = new FlowExecutionAppConnLinkisJob + linkisJob.setJobProps(this.jobProps) + linkisJob + } + } + + override protected def fillJobInfo(job: Job): Unit = { + if(StringUtils.isNotEmpty(jobProps.get(FlowExecutionEntranceConfiguration.COMMAND))) + job.setCode(jobProps.get(FlowExecutionEntranceConfiguration.COMMAND)) + job.setParams(new util.HashMap[String,AnyRef]()) + val runtimeMap = new util.HashMap[String, AnyRef]() + //update by peaceWong add contextID to runtimeMap + if (null != jobProps.get(FlowExecutionEntranceConfiguration.CONTEXT_ID)){ + runtimeMap.put(FlowExecutionEntranceConfiguration.CONTEXT_ID, jobProps.get(FlowExecutionEntranceConfiguration.CONTEXT_ID)) + } + runtimeMap.put(FlowExecutionEntranceConfiguration.NODE_NAME,jobProps.get(FlowExecutionEntranceConfiguration.JOB_ID)) + runtimeMap.put(FlowExecutionEntranceConfiguration.WORKSPACE, jobProps.get(FlowExecutionEntranceConfiguration.WORKSPACE)) + runtimeMap.put(DSSCommonUtils.DSS_LABELS_KEY, jobProps.get(DSSCommonUtils.DSS_LABELS_KEY)) + job.setRuntimeParams(runtimeMap) + } + + override protected def fillLinkisJobInfo(linkisJob: LinkisJob): Unit = { + this.node.getDSSNode.getParams.get(FlowExecutionEntranceConfiguration.NODE_CONFIGURATION_KEY) match { + case configuration:util.Map[String, AnyRef] => + linkisJob.setConfiguration(configuration) + case _ => + } + this.node.getDSSNode.getParams.remove(FlowExecutionEntranceConfiguration.FLOW_VAR_MAP) match { + case flowVar:util.Map[String, AnyRef] => + linkisJob.setVariables(flowVar) + case _ => + } + linkisJob.setSource(getSource) + } + + override protected def fillCommonLinkisJobInfo(linkisAppConnJob: CommonLinkisJob): Unit = { + linkisAppConnJob.setJobResourceList(FlowExecutionUtils.resourcesAdaptation(this.node.getDSSNode.getResources)) + this.node.getDSSNode.getParams.remove(FlowExecutionEntranceConfiguration.PROJECT_RESOURCES) match { + case projectResources:util.List[Resource] => + linkisAppConnJob.setProjectResourceList(FlowExecutionUtils.resourcesAdaptation(projectResources)) + case _ => + } + this.node.getDSSNode.getParams.remove(FlowExecutionEntranceConfiguration.FLOW_RESOURCES) match { + case flowResources:util.HashMap[String, util.List[BMLResource]] => + linkisAppConnJob.setFlowNameAndResources(flowResources) + case _ => + } + } + + + private def getSource: util.Map[String, String] = { + val source = new util.HashMap[String, String]() + source.put("projectName", jobProps.get(FlowExecutionEntranceConfiguration.PROJECT_NAME)) + source.put("flowName", jobProps.get(FlowExecutionEntranceConfiguration.FLOW_NAME)) + source.put("nodeName", jobProps.get(FlowExecutionEntranceConfiguration.JOB_ID)) + source + } + + + } +} diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/DefaultNodeRunner.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/DefaultNodeRunner.scala similarity index 82% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/DefaultNodeRunner.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/DefaultNodeRunner.scala index fbd13c849..42fcfa22c 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/DefaultNodeRunner.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/DefaultNodeRunner.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,23 +18,22 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.node import java.util -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration -import com.webank.wedatasphere.dss.linkis.node.execution.job.{JobTypeEnum, LinkisJob} import com.webank.wedatasphere.dss.flow.execution.entrance.listener.NodeRunnerListener import com.webank.wedatasphere.dss.flow.execution.entrance.log.FlowExecutionLog import com.webank.wedatasphere.dss.flow.execution.entrance.node.NodeExecutionState.NodeExecutionState +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration import com.webank.wedatasphere.dss.linkis.node.execution.execution.impl.LinkisNodeExecutionImpl +import com.webank.wedatasphere.dss.linkis.node.execution.job.{JobTypeEnum, LinkisJob} import com.webank.wedatasphere.dss.linkis.node.execution.listener.LinkisExecutionListener +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode import com.webank.wedatasphere.linkis.common.exception.ErrorException import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -/** - * Created by johnnwang on 2019/11/5. - */ + class DefaultNodeRunner extends NodeRunner with Logging { - private var node: SchedulerNode = _ + private var node: WorkflowNode = _ private var linkisJob: LinkisJob = _ @@ -53,9 +51,9 @@ class DefaultNodeRunner extends NodeRunner with Logging { private var lastGetStatusTime: Long = 0 - override def getNode: SchedulerNode = this.node + override def getNode: WorkflowNode = this.node - def setNode(schedulerNode: SchedulerNode): Unit = { + def setNode(schedulerNode: WorkflowNode): Unit = { this.node = schedulerNode } @@ -103,11 +101,11 @@ class DefaultNodeRunner extends NodeRunner with Logging { override def run(): Unit = { info(s"start to run node of ${node.getName}") try { - val jobProps = node.getDssNode.getParams.remove(FlowExecutionEntranceConfiguration.PROPS_MAP) match { + val jobProps = node.getDSSNode.getParams.remove(FlowExecutionEntranceConfiguration.PROPS_MAP) match { case propsMap: util.Map[String, String] => propsMap case _ => new util.HashMap[String, String]() } - this.linkisJob = AppJointJobBuilder.builder().setNode(node).setJobProps(jobProps).build.asInstanceOf[LinkisJob] + this.linkisJob = AppConnJobBuilder.builder().setNode(node).setJobProps(jobProps).build.asInstanceOf[LinkisJob] this.linkisJob.setLogObj(new FlowExecutionLog(this)) //set start time this.setStartTime(System.currentTimeMillis()) @@ -116,15 +114,9 @@ class DefaultNodeRunner extends NodeRunner with Logging { this.transitionState(NodeExecutionState.Succeed) return } - + this.linkisJob.getJobProps.put(LinkisJobExecutionConfiguration.LINKIS_VERSION_KEY, LinkisJobExecutionConfiguration.LINKIS_DEFAULT_VERSION.getValue) LinkisNodeExecutionImpl.getLinkisNodeExecution.runJob(this.linkisJob) - info(s"start to run node of ${node.getName}") - /*LinkisNodeExecutionImpl.getLinkisNodeExecution.waitForComplete(this.linkisJob) - val listener = LinkisNodeExecutionImpl.getLinkisNodeExecution.asInstanceOf[LinkisExecutionListener] - val toState = LinkisNodeExecutionImpl.getLinkisNodeExecution.getState(this.linkisJob) - listener.onStatusChanged(getStatus.toString, toState, this.linkisJob) - this.transitionState(NodeExecutionState.withName(toState)) - info(s"Finished to execute node of ${node.getName}")*/ + info(s"Finished to run node of ${node.getName}") } catch { case t: Throwable => warn(s"Failed to execute node of ${node.getName}", t) diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeExecutionState.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeExecutionState.scala similarity index 91% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeExecutionState.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeExecutionState.scala index 2df6ff320..d44b7dfc4 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeExecutionState.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeExecutionState.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -18,9 +17,6 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.node -/** - * Created by peacewong on 2019/11/5. - */ object NodeExecutionState extends Enumeration { type NodeExecutionState = Value diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeRunner.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeRunner.scala similarity index 91% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeRunner.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeRunner.scala index f0a40fd57..686498666 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeRunner.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/node/NodeRunner.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,18 +16,16 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.node -import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob +import java.util.concurrent.Future + import com.webank.wedatasphere.dss.flow.execution.entrance.FlowContext import com.webank.wedatasphere.dss.flow.execution.entrance.listener.NodeRunnerListener import com.webank.wedatasphere.dss.flow.execution.entrance.node.NodeExecutionState.NodeExecutionState +import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import java.util.concurrent.Future -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode -/** - * Created by peacewong on 2019/11/5. - */ abstract class NodeRunner extends Runnable with Logging{ private[flow] var future: Future[_] = _ @@ -43,9 +40,9 @@ abstract class NodeRunner extends Runnable with Logging{ this.flowContext = flowContext } - def getNode:SchedulerNode + def getNode: WorkflowNode - def setNode(node: SchedulerNode):Unit + def setNode(node: WorkflowNode):Unit def getLinkisJob: LinkisJob diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolver.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolver.scala similarity index 84% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolver.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolver.scala index 424d1424e..0f8b898db 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolver.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolver.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,10 +18,7 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.resolver import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob -/** - * created by chaogefeng on 2019/11/5 10:20 - * Description: - */ + trait FlowDependencyResolver { def resolvedFlow(flowJob:FlowEntranceJob) } diff --git a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolverImpl.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolverImpl.scala similarity index 90% rename from dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolverImpl.scala rename to dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolverImpl.scala index ecd93ffd1..b114e47e4 100644 --- a/dss-flow-execution-entrance/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolverImpl.scala +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/resolver/FlowDependencyResolverImpl.scala @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,18 +18,14 @@ package com.webank.wedatasphere.dss.flow.execution.entrance.resolver import java.util -import com.webank.wedatasphere.dss.flow.execution.entrance.{FlowContext} +import com.webank.wedatasphere.dss.flow.execution.entrance.FlowContext import com.webank.wedatasphere.dss.flow.execution.entrance.job.FlowEntranceJob - -import scala.collection.JavaConversions._ import com.webank.wedatasphere.linkis.common.utils.Logging import org.springframework.stereotype.Component +import scala.collection.JavaConversions._ + -/** - * created by chaogefeng on 2019/11/5 10:24 - * Description: - */ @Component class FlowDependencyResolverImpl extends FlowDependencyResolver with Logging { override def resolvedFlow(flowJob: FlowEntranceJob) = { diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/utils/FlowExecutionUtils.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/utils/FlowExecutionUtils.scala new file mode 100644 index 000000000..8fbe22f4f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-flow-execution-server/src/main/scala/com/webank/wedatasphere/dss/flow/execution/entrance/utils/FlowExecutionUtils.scala @@ -0,0 +1,112 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.flow.execution.entrance.utils + +import java.lang.reflect.Type +import java.util.Date +import java.{lang, util} + +import com.google.gson._ +import com.webank.wedatasphere.dss.common.entity.Resource +import com.webank.wedatasphere.dss.flow.execution.entrance.conf.FlowExecutionEntranceConfiguration +import com.webank.wedatasphere.dss.flow.execution.entrance.exception.FlowExecutionErrorException +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration +import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode +import com.webank.wedatasphere.linkis.governance.common.entity.job.JobRequest +import com.webank.wedatasphere.linkis.governance.common.entity.task.RequestPersistTask +import com.webank.wedatasphere.linkis.manager.label.entity.Label +import com.webank.wedatasphere.linkis.manager.label.entity.engine.CodeLanguageLabel +import com.webank.wedatasphere.linkis.manager.label.utils.LabelUtil + +import scala.collection.JavaConversions._ +import scala.collection.JavaConverters.asScalaBufferConverter + + +object FlowExecutionUtils { + + def isSkippedNode(node: WorkflowNode):Boolean = { + FlowExecutionEntranceConfiguration.SKIP_NODES.getValue.split(",").exists(_.equalsIgnoreCase(node.getNodeType)) + } + + def getRunTypeLabel(labels: util.List[Label[_]]): CodeLanguageLabel = { + labels.find { + case label: CodeLanguageLabel => true + case _ => false + }.map(_.asInstanceOf[CodeLanguageLabel]).getOrElse(throw new FlowExecutionErrorException(90106, "Cannot find runType label.")) + } + + def isSignalNode(jobType: String) : Boolean = { + FlowExecutionEntranceConfiguration.SIGNAL_NODES.getValue.split(",").exists(_.equalsIgnoreCase(jobType)) + } + + def isAppConnJob(engineType: String): Boolean = LinkisJobExecutionConfiguration.APPCONN.equalsIgnoreCase(engineType) + + + def resourcesAdaptation(resources: util.List[Resource]): util.ArrayList[BMLResource] = { + val bmlResources = new util.ArrayList[BMLResource]() + for (resource <- resources){ + bmlResources.add(resourceConvertToBMLResource(resource)) + } + bmlResources + } + + def resourceConvertToBMLResource(resource: Resource):BMLResource = { + val bmlResource = new BMLResource() + bmlResource.setFileName(resource.getFileName) + bmlResource.setResourceId(resource.getResourceId) + bmlResource.setVersion(resource.getVersion) + bmlResource + } + + + def jobRequest2RequestPersistTask(jobReq: JobRequest): RequestPersistTask = { + if (null == jobReq) return null + val persistTask = new RequestPersistTask + persistTask.setTaskID(jobReq.getId) + persistTask.setExecId(jobReq.getReqId) + // jobHistory.setPriority(jobReq.getProgress) + persistTask.setSubmitUser(jobReq.getSubmitUser) + persistTask.setUmUser(jobReq.getExecuteUser) + persistTask.setSource(jobReq.getSource) + if (null != jobReq.getLabels) { + val labelMap = new util.HashMap[String, String](jobReq.getLabels.size()) + jobReq.getLabels.asScala.map(l => l.getLabelKey -> l.getStringValue).foreach(kv => labelMap.put(kv._1, kv._2)) + persistTask.setLabels(jobReq.getLabels) + } + if (null != jobReq.getParams) persistTask.setParams(jobReq.getParams) + persistTask.setProgress(jobReq.getProgress.toFloat) + persistTask.setStatus(jobReq.getStatus) + persistTask.setLogPath(jobReq.getLogPath) + persistTask.setErrCode(jobReq.getErrorCode) + persistTask.setErrDesc(jobReq.getErrorDesc) + if (null != jobReq.getCreatedTime) persistTask.setCreatedTime(new Date(jobReq.getCreatedTime.getTime)) + if (null != jobReq.getUpdatedTime) persistTask.setUpdatedTime(new Date(jobReq.getUpdatedTime.getTime)) + persistTask.setInstance(jobReq.getInstances) +// if (null != jobReq.getMetrics) persistTask.set(gson.toJson(jobReq.getMetrics)) + val engineType = LabelUtil.getEngineType(jobReq.getLabels) + persistTask.setEngineType(engineType) + persistTask + } + + implicit val gson = new GsonBuilder().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").serializeNulls + .registerTypeAdapter(classOf[java.lang.Double], new JsonSerializer[java.lang.Double] { + override def serialize(t: lang.Double, `type`: Type, jsonSerializationContext: JsonSerializationContext): JsonElement = + if(t == t.longValue()) new JsonPrimitive(t.longValue()) else new JsonPrimitive(t) + }).create + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/pom.xml new file mode 100644 index 000000000..30ae31bfa --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/pom.xml @@ -0,0 +1,78 @@ + + + + + 4.0.0 + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + + dss-linkis-node-execution + + + + com.webank.wedatasphere.linkis + linkis-computation-client + ${linkis.version} + + + + com.webank.wedatasphere.linkis + linkis-protocol + ${linkis.version} + + + + com.webank.wedatasphere.linkis + linkis-storage-script-dev-client + ${linkis.version} + + + + com.webank.wedatasphere.linkis + linkis-label-common + ${linkis.version} + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.2 + + 1.8 + 1.8 + UTF-8 + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/conf/LinkisJobExecutionConfiguration.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/conf/LinkisJobExecutionConfiguration.java new file mode 100644 index 000000000..15f425756 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/conf/LinkisJobExecutionConfiguration.java @@ -0,0 +1,93 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.conf; + + + +import com.webank.wedatasphere.linkis.common.conf.CommonVars; +import com.webank.wedatasphere.linkis.common.conf.Configuration; +import java.util.Map; + + +public class LinkisJobExecutionConfiguration { + + + public static final String LINKIS_TYPE = "linkistype"; + + public static final String APPCONN = "appconn"; + + public static final String DSS_LABELS_KEY = "labels"; + + public static final String PROJECT_PREFIX = "project"; + public static final String JOB_PREFIX = "job"; + + public static final String FLOW_VARIABLE_PREFIX = "flow.variable."; + + public static final String NODE_CONF_PREFIX = "node.conf."; + + public static final String CONF_STARTUP = "startup"; + public static final String CONF_RUNTIME = "runtime"; + public static final String CONF_SPECIAL = "special"; + + public static final String RESOURCES_NAME = ".resources"; + + public final static String LINKIS_VERSION_KEY = "linkis.version"; + + public static final String LINKIS_SUBMIT_USER = "wds.dss.workflow.submit.user"; + + public static final String LINKIS_CONTROL_EMPTY_NODE = "linkis.control.empty"; + + public static final String FLOW_CONTEXTID = "wds.linkis.flow.contextID"; + + public final static CommonVars JOB_DEFAULT_TYPE = CommonVars.apply("wds.linkis.flow.default.job.type","linkis"); + + public final static CommonVars LINKIS_DEFAULT_TYPE = CommonVars.apply("wds.linkis.flow.default.linkis.type","spark.sql"); + + public final static CommonVars LINKIS_CONNECTION_TIMEOUT = CommonVars.apply("wds.linkis.flow.connection.timeout",300000); + + public final static CommonVars LINKIS_JOB_REQUEST_STATUS_TIME = CommonVars.apply("wds.linkis.job.status.timeout",3000); + + public final static CommonVars LINKIS_ADMIN_USER = CommonVars.apply("wds.linkis.client.flow.adminuser","ws"); + + + public final static CommonVars LINKIS_AUTHOR_USER_TOKEN = CommonVars.apply("wds.linkis.client.flow.author.user.token","WS-AUTH"); + + public final static CommonVars LINKIS_JOB_CREATOR = CommonVars.apply("wds.linkis.flow.job.creator","nodeexecution"); + + public final static CommonVars LINKIS_URL = CommonVars.apply("wds.linkis.gateway.url.v0", Configuration.getGateWayURL()); + + public final static CommonVars LINKIS_URL_1_X = CommonVars.apply("wds.linkis.gateway.url.v1", Configuration.getGateWayURL()); + + public final static CommonVars LINKIS_API_VERSION = CommonVars.apply("wds.linkis.server.version","v1"); + + public final static CommonVars RESULT_PRINT_SIZE = CommonVars.apply("wds.linkis.result.print.size",10); + + + public final static CommonVars LOG_SIZE = CommonVars.apply("wds.linkis.log.size",-1); + public final static CommonVars LOG_ARRAY_LEN = CommonVars.apply("wds.linkis.log.array.len",4); + + public final static CommonVars REQUEST_MAX_RETRY_TIME = CommonVars.apply("wds.linkis.log.retry.time",10); + + //兼容老版本 + public static final CommonVars LINKIS_DEFAULT_VERSION = CommonVars.apply("wds.dss.workflow.execution.linkis.version", "1.0.0"); + + public static boolean isLinkis1_X(Map props) { + return props.getOrDefault(LinkisJobExecutionConfiguration.LINKIS_VERSION_KEY,"") + .startsWith("1."); + } + +} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/BMLResource.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/BMLResource.java similarity index 90% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/BMLResource.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/BMLResource.java index 24e6cd591..f9b6fbe75 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/BMLResource.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/BMLResource.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,9 +16,7 @@ package com.webank.wedatasphere.dss.linkis.node.execution.entity; -/** - * Created by peacewong on 2019/9/21. - */ + public class BMLResource { private String resourceId; diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/ContextInfo.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/ContextInfo.java similarity index 90% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/ContextInfo.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/ContextInfo.java index 240bf7cc4..5bef46159 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/ContextInfo.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/entity/ContextInfo.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,9 +16,7 @@ package com.webank.wedatasphere.dss.linkis.node.execution.entity; -/** - * Created by peacewong on 2019/9/26. - */ + public class ContextInfo { diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionErrorException.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionErrorException.java similarity index 87% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionErrorException.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionErrorException.java index 7db0810ad..f4916634d 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionErrorException.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionErrorException.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -20,9 +19,7 @@ import com.webank.wedatasphere.linkis.common.exception.ErrorException; -/** - * Created by peacewong on 2019/9/19. - */ + public class LinkisJobExecutionErrorException extends ErrorException { public LinkisJobExecutionErrorException(int errCode, String desc) { super(errCode, desc); diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionWarnException.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionWarnException.java similarity index 89% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionWarnException.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionWarnException.java index 5e77e97a5..40601b17a 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionWarnException.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/exception/LinkisJobExecutionWarnException.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,9 +18,7 @@ import com.webank.wedatasphere.linkis.common.exception.WarnException; -/** - * Created by enjoyyin on 2019/10/29. - */ + public class LinkisJobExecutionWarnException extends WarnException { public LinkisJobExecutionWarnException(int errCode, String desc) { diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/LinkisNodeExecution.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/LinkisNodeExecution.java similarity index 86% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/LinkisNodeExecution.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/LinkisNodeExecution.java index 73acc8088..4c5b4b771 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/LinkisNodeExecution.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/LinkisNodeExecution.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -18,11 +17,10 @@ package com.webank.wedatasphere.dss.linkis.node.execution.execution; import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import java.io.Closeable; -/** - * Created by peacewong on 2019/11/2. - */ -public interface LinkisNodeExecution { + +public interface LinkisNodeExecution extends Closeable { void runJob(Job job) throws Exception; diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/impl/LinkisNodeExecutionImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/impl/LinkisNodeExecutionImpl.java new file mode 100644 index 000000000..e3ca11a56 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/execution/impl/LinkisNodeExecutionImpl.java @@ -0,0 +1,351 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.execution.impl; + +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; +import com.webank.wedatasphere.dss.linkis.node.execution.execution.LinkisNodeExecution; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.listener.LinkisExecutionListener; +import com.webank.wedatasphere.dss.linkis.node.execution.parser.CodeParser; +import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParamsParser; +import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParser; +import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobRuntimeParamsParser; +import com.webank.wedatasphere.dss.linkis.node.execution.service.LinkisURLService; +import com.webank.wedatasphere.dss.linkis.node.execution.service.impl.BuildJobActionImpl; +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.parser.CodeParser; +import com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParser; +import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils; +import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisUjesClientUtils; +import com.webank.wedatasphere.linkis.common.exception.LinkisException; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import com.webank.wedatasphere.linkis.ujes.client.UJESClient; +import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; +import com.webank.wedatasphere.linkis.ujes.client.request.JobSubmitAction; +import com.webank.wedatasphere.linkis.ujes.client.request.OpenLogAction; +import com.webank.wedatasphere.linkis.ujes.client.request.ResultSetAction; +import com.webank.wedatasphere.linkis.ujes.client.response.JobInfoResult; +import com.webank.wedatasphere.linkis.ujes.client.response.JobLogResult; +import com.webank.wedatasphere.linkis.ujes.client.response.OpenLogResult; +import org.apache.commons.lang.StringUtils; +import scala.tools.nsc.settings.Final; + +import java.text.MessageFormat; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang.StringUtils; + + +public class LinkisNodeExecutionImpl implements LinkisNodeExecution , LinkisExecutionListener { + + private static LinkisNodeExecution linkisExecution = new LinkisNodeExecutionImpl(); + + private LinkisNodeExecutionImpl() { + registerJobParser(new CodeParser()); + registerJobParser(new JobRuntimeParamsParser()); + registerJobParser(new JobParamsParser()); + } + + public static LinkisNodeExecution getLinkisNodeExecution() { + return linkisExecution; + } + + private final Map clientMap = new HashMap<>(); + + private ArrayList jobParsers = new ArrayList<>(); + + protected UJESClient getClient(Job job) { + Map props =job.getJobProps(); + String linkisUrl = LinkisURLService.Factory.getLinkisURLService().getLinkisURL(job); + if(clientMap.containsKey(linkisUrl)) { + return clientMap.get(linkisUrl); + } + synchronized (clientMap) { + if(!clientMap.containsKey(linkisUrl)) { + UJESClient client = LinkisUjesClientUtils.getUJESClient( + linkisUrl, + LinkisJobExecutionConfiguration.LINKIS_ADMIN_USER.getValue(props), + LinkisJobExecutionConfiguration.LINKIS_AUTHOR_USER_TOKEN.getValue(props), + props); + clientMap.put(linkisUrl, client); + job.getLogObj().info("Create a new Linkis client by " + linkisUrl); + return client; + } + } + return clientMap.get(linkisUrl); + } + + public void registerJobParser(JobParser jobParser) { + this.jobParsers.add(jobParser); + } + + @Override + public void runJob(Job job) throws Exception { + // job parser + for(JobParser parser : jobParsers){ + parser.parseJob(job); + } + + Map props =job.getJobProps(); + if(LinkisJobExecutionConfiguration.isLinkis1_X(props)) { + JobSubmitAction submitAction = BuildJobActionImpl.getbuildJobAction().getSubmitAction(job); + job.setJobExecuteResult(getClient(job).submit(submitAction)); + }else{ + //兼容0.X版本的任务提交方式 + JobExecuteAction jobAction = BuildJobActionImpl.getbuildJobAction().getJobAction(job); + job.setJobExecuteResult(getClient(job).execute(jobAction)); + } + + job.getLogObj().info("<---------------Start to execute job--------------->"); + job.getLogObj().info("Task id is:"+ job.getJobExecuteResult().getTaskID()); + job.getLogObj().info("Exec id is:"+ job.getJobExecuteResult().getExecID()); + } + + @Override + public String getState(Job job) { + return getClient(job).getJobInfo(job.getJobExecuteResult()).getJobStatus(); + } + + @Override + public String getLog(Job job) { + + JobInfoResult jobInfo =getClient(job).getJobInfo(job.getJobExecuteResult()); + + List logArray = null; + //只有job执行失败了，才打印全部日志，否则只打印linkis运行时缓存日志 + if(jobInfo.isCompleted() && jobInfo.isFailed()){ + try { + logArray = Arrays.asList(queryPersistedLogAll(job).getLog()); + } catch (LinkisJobExecutionErrorException e) { + job.getLogObj().error("Get full log failed:"+e.getMessage()); + } + }else if(jobInfo.isRunning()){ + JobLogResult jobLogResult = getClient(job) + .log(job.getJobExecuteResult(), + job.getLogFromLine(), + LinkisJobExecutionConfiguration.LOG_SIZE.getValue()); + + job.setLogFromLine(jobLogResult.fromLine()); + logArray = jobLogResult.getLog(); + }else { + job.getLogObj().info("Job run is completed and the cache log can not be printed "); + } + + if (logArray != null && logArray.size() + >= LinkisJobExecutionConfiguration.LOG_ARRAY_LEN.getValue() + && StringUtils.isNotEmpty(logArray.get(3))) { + return logArray.get(3); + } + return null; + } + + + + + + public OpenLogResult queryPersistedLogAll(Job job) throws LinkisJobExecutionErrorException { + String taskID = job.getJobExecuteResult().getTaskID(); + String user = job.getUser(); + JobInfoResult jobInfo = getClient(job).getJobInfo(job.getJobExecuteResult()); + String logPath = jobInfo.getRequestPersistTask().getLogPath(); + int retryCnt = 0; + final int MAX_RETRY_TIMES = LinkisJobExecutionConfiguration.REQUEST_MAX_RETRY_TIME.getValue(); + OpenLogResult openLogResult = null; + + int backCnt = 0; + final int MAX_BACK_TIMES = 3; + + while (retryCnt++ < MAX_RETRY_TIMES) { + try { + openLogResult = getClient(job).openLog(OpenLogAction.newBuilder().setLogPath(logPath).setProxyUser(user).build()); + job.getLogObj().info("persisted-log-result:" + LinkisJobExecutionUtils.gson.toJson(openLogResult)); + if (openLogResult == null || + 0 != openLogResult.getStatus() || + StringUtils.isBlank(openLogResult.getLog()[LinkisJobExecutionUtils.IDX_FOR_LOG_TYPE_ALL])) { + String reason; + if (openLogResult == null) { + reason = "OpenLogResult is null"; + } else if (0 != openLogResult.getStatus()) { + reason = "server returns non-zero status-code"; + } else { + reason = "server returns empty log"; + } + String msg = MessageFormat.format("Get log from openLog failed. retry time : {0}/{1}. taskID={2}. Reason: {3}", retryCnt, MAX_RETRY_TIMES, taskID, reason); + job.getLogObj().warn(msg); + if (retryCnt >= MAX_RETRY_TIMES) { + if (backCnt >= MAX_BACK_TIMES) { + msg = MessageFormat.format("Get log from openLog failed. Retry exhausted. taskID={0}, Reason: {1}", job.getJobExecuteResult().getTaskID(), reason); + throw new LinkisJobExecutionErrorException(100079,msg); + } else { + backCnt++; + retryCnt = 0; + Utils.sleepQuietly(LinkisJobExecutionConfiguration.LINKIS_JOB_REQUEST_STATUS_TIME.getValue(job.getJobProps()));;//wait 10s and try again + } + } + } else { + break; + } + } catch (Exception e) { + String msg = MessageFormat.format("Get log from openLog failed. retry time : {0}/{1}", retryCnt, MAX_RETRY_TIMES); + if (e instanceof LinkisException) { + msg += " " + e.toString(); + } + job.getLogObj().warn(msg, e); + if (retryCnt >= MAX_RETRY_TIMES) { + if (backCnt >= MAX_BACK_TIMES) { + throw new LinkisJobExecutionErrorException(100080, "Get log from openLog failed. Retry exhausted. taskID=" + taskID); + } else { + backCnt++; + retryCnt = 0; + Utils.sleepQuietly(LinkisJobExecutionConfiguration.LINKIS_JOB_REQUEST_STATUS_TIME.getValue(job.getJobProps()));//wait 10s and try again + } + } + } + Utils.sleepQuietly(LinkisJobExecutionConfiguration.LINKIS_JOB_REQUEST_STATUS_TIME.getValue(job.getJobProps()));; + } + return openLogResult; + } + + + + + @Override + public void waitForComplete(Job job) throws Exception { + JobInfoResult jobInfo = getClient(job).getJobInfo(job.getJobExecuteResult()); + int count = 0; + while (!jobInfo.isCompleted()) { + double progress = -1; + try{ + progress = this.getProgress(job); + }catch(Exception e){ + //ignore + } + if (progress >= 0){ + job.getLogObj().info("Update Progress info:" + progress); + } + JobInfoResult oldJobInfo = jobInfo; + try{ + jobInfo = getClient(job).getJobInfo(job.getJobExecuteResult()); + }catch(Throwable e){ + jobInfo = oldJobInfo; + count += 1; + job.getLogObj().error("不能获取到正确的状态，计数 count = " + count); + //两分钟内获取不到,就认为不行，因为这个时候我应该是重启完成了 + if (count == 40) { + job.getLogObj().info("超过40次不能获取状态，应该是linkis不能获取到正常信息，判断任务失败"); + throw new LinkisJobExecutionErrorException(90101, "Failed to execute Job: " + e.getMessage()); + } + } + Utils.sleepQuietly(LinkisJobExecutionConfiguration.LINKIS_JOB_REQUEST_STATUS_TIME.getValue(job.getJobProps())); + printJobLog(job); + } + if (!jobInfo.isSucceed()) { + //printJobLog(job); + throw new LinkisJobExecutionErrorException(90101, "Failed to execute Job: " + jobInfo.getMessage()); + } + } + + private void printJobLog(Job job){ + String log = null; + try{ + log = this.getLog(job); + }catch(Exception e){ + job.getLogObj().warn("failed to get log info", e); + } + if (StringUtils.isEmpty(log)){ + return; + } + Arrays.stream(log.split("\n")).forEach(l -> { + if (l != null) { + if(l.contains("ERROR")) { + job.getLogObj().error(l); + }else{ + job.getLogObj().info(l); + } + } + }); + + } + + @Override + public void cancel(Job job) throws Exception { + getClient(job).kill(job.getJobExecuteResult()); + } + + @Override + public double getProgress(Job job) { + return getClient(job).progress(job.getJobExecuteResult()).getProgress(); + } + + @Override + public Boolean isCompleted(Job job) { + return getClient(job).getJobInfo(job.getJobExecuteResult()).isCompleted(); + } + + @Override + public int getResultSize(Job job) { + JobInfoResult jobInfo = getClient(job).getJobInfo(job.getJobExecuteResult()); + + job.getLogObj().info("JobInfo result location is "+jobInfo.getRequestPersistTask().getResultLocation()); + + job.getLogObj().info("JobInfo user location is "+jobInfo.getRequestPersistTask().getUmUser()); + //解决azkaban 执行报 user is need问题 + jobInfo.getRequestPersistTask().setUmUser(job.getUser()); + if (jobInfo.isSucceed()) { + String[] resultSetList = jobInfo.getResultSetList(getClient(job)); + if (resultSetList != null && resultSetList.length > 0) { + return resultSetList.length; + } + } + return 0; + } + + @Override + public String getResult(Job job, int index, int maxSize) { + String resultContent = null; + JobInfoResult jobInfo = getClient(job).getJobInfo(job.getJobExecuteResult()); + String[] resultSetList = jobInfo.getResultSetList(getClient(job)); + if (resultSetList != null && resultSetList.length > 0) { + Object fileContent = getClient(job).resultSet(ResultSetAction.builder() + .setPath(resultSetList[index]) + .setUser(job.getJobExecuteResult().getUser()) + .setPageSize(maxSize).build()).getFileContent(); + if (fileContent instanceof ArrayList) { + ArrayList> resultSetRow = (ArrayList>) fileContent; + resultContent = StringUtils.join(resultSetRow, "\n"); + } else { + resultContent = fileContent.toString(); + } + } + return resultContent; + } + + @Override + public void onStatusChanged(String fromState, String toState, Job job) { + } + + @Override + public void close() { + clientMap.values().forEach(IOUtils::closeQuietly); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractAppConnLinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractAppConnLinkisJob.java new file mode 100644 index 000000000..aa0755acb --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractAppConnLinkisJob.java @@ -0,0 +1,192 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.job; + +import com.webank.wedatasphere.dss.linkis.node.execution.log.LinkisJobExecutionLog; +import com.webank.wedatasphere.linkis.ujes.client.response.JobExecuteResult; + +import java.util.Map; + + +public abstract class AbstractAppConnLinkisJob extends AppConnLinkisJob { + + private Map jobProps; + private Map source; + + private JobTypeEnum jobType; + + private Map variables; + + private Map configuration; + + private String code; + + private String engineType; + + private String runType; + + + + private Map params; + + private Map runtimeParams; + + private JobExecuteResult jobExecuteResult; + + private LinkisJobExecutionLog logObj; + + private int logFromLine; + + + + + + + @Override + public Map getSource() { + return this.source; + } + + @Override + public JobTypeEnum getJobType() { + return this.jobType; + } + + + @Override + public Map getVariables() { + return this.variables; + } + + @Override + public Map getConfiguration() { + return this.configuration; + } + + @Override + public String getCode() { + return this.code; + } + + @Override + public void setCode(String code) { + this.code = code; + } + + @Override + public String getEngineType() { + return this.engineType; + } + + @Override + public String getRunType() { + return this.runType; + } + + + + @Override + public Map getParams() { + return this.params; + } + + @Override + public Map getRuntimeParams() { + return this.runtimeParams; + } + + @Override + public JobExecuteResult getJobExecuteResult() { + return this.jobExecuteResult; + } + + @Override + public void setJobExecuteResult(JobExecuteResult jobExecuteResult) { + this.jobExecuteResult = jobExecuteResult; + } + + @Override + public Map getJobProps() { + return this.jobProps; + } + + @Override + public void setJobProps(Map jobProps) { + this.jobProps = jobProps; + } + + @Override + public LinkisJobExecutionLog getLogObj() { + return this.logObj; + } + + @Override + public int getLogFromLine() { + return this.logFromLine; + } + + @Override + public void setLogFromLine(int index) { + this.logFromLine = index; + } + + + @Override + public void setSource(Map source) { + this.source = source; + } + + @Override + public void setJobType(JobTypeEnum jobType) { + this.jobType = jobType; + } + + @Override + public void setVariables(Map variables) { + this.variables = variables; + } + + @Override + public void setConfiguration(Map configuration) { + this.configuration = configuration; + } + + @Override + public void setEngineType(String engineType) { + this.engineType = engineType; + } + + @Override + public void setRunType(String runType) { + this.runType = runType; + } + + @Override + public void setParams(Map params) { + this.params = params; + } + + @Override + public void setRuntimeParams(Map runtimeParams) { + this.runtimeParams = runtimeParams; + } + + @Override + public void setLogObj(LinkisJobExecutionLog logObj) { + this.logObj = logObj; + } +} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractCommonLinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractCommonLinkisJob.java similarity index 97% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractCommonLinkisJob.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractCommonLinkisJob.java index c7f74f234..a36624e04 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractCommonLinkisJob.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AbstractCommonLinkisJob.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -25,9 +24,7 @@ import java.util.List; import java.util.Map; -/** - * Created by peacewong on 2019/11/3. - */ + public abstract class AbstractCommonLinkisJob extends CommonLinkisJob { private Map jobProps; diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AppConnLinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AppConnLinkisJob.java new file mode 100644 index 000000000..48e18a546 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/AppConnLinkisJob.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.job; + + +public abstract class AppConnLinkisJob extends LinkisJob { +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Builder.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Builder.java new file mode 100644 index 000000000..acf5f4890 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Builder.java @@ -0,0 +1,89 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.job; + +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; +import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils; +import org.apache.commons.lang.StringUtils; + + +public abstract class Builder { + + protected abstract String getJobType(); + + protected abstract LinkisJob creatLinkisJob(boolean isLinkisType); + + protected abstract void fillJobInfo(Job job); + + protected abstract void fillLinkisJobInfo(LinkisJob linkisJob); + + protected abstract void fillCommonLinkisJobInfo(CommonLinkisJob linkisAppConnJob); + + + public Job build() throws Exception { + + LinkisJob job = null; + String jobType = getJobType(); + String[] jobTypeSplit = jobType.split("\\."); + if (jobTypeSplit.length < 3) { + throw new LinkisJobExecutionErrorException(90100, "This is not Linkis job type,this jobtype is " + jobType); + } + String engineType = jobTypeSplit[1]; + //delete linkis.engineType + String runType = StringUtils.substringAfterLast(jobType, jobTypeSplit[0] + "." + jobTypeSplit[1] + "."); + + if (LinkisJobExecutionConfiguration.LINKIS_CONTROL_EMPTY_NODE.equalsIgnoreCase(jobType)) { + job = new AbstractCommonLinkisJob() { + @Override + public String getSubmitUser() { + return null; + } + + @Override + public String getUser() { + return null; + } + + @Override + public String getJobName() { + return null; + } + }; + + job.setJobType(JobTypeEnum.EmptyJob); + return job; + } + //update by peaceWong + if (LinkisJobExecutionUtils.isCommonAppConnJob(engineType)) { + job = creatLinkisJob(false); + job.setJobType(JobTypeEnum.CommonJob); + } else { + job = creatLinkisJob(true); + job.setJobType(JobTypeEnum.CommonJob); + fillCommonLinkisJobInfo((CommonLinkisJob) job); + } + + job.setEngineType(engineType); + job.setRunType(runType); + fillJobInfo(job); + fillLinkisJobInfo(job); + + return job; + } + +} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/CommonLinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/CommonLinkisJob.java similarity index 91% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/CommonLinkisJob.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/CommonLinkisJob.java index 229ee2862..cac95a127 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/CommonLinkisJob.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/CommonLinkisJob.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -23,9 +22,7 @@ import java.util.List; import java.util.Map; -/** - * Created by peacewong on 2019/11/2. - */ + public abstract class CommonLinkisJob extends LinkisJob { public abstract List getJobResourceList(); diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Job.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Job.java similarity index 93% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Job.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Job.java index 1811ad105..e18a402e8 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Job.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/Job.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -22,9 +21,6 @@ import java.util.Map; -/** - * Created by peacewong on 2019/10/15. - */ public interface Job { String getCode(); diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobTypeEnum.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobTypeEnum.java similarity index 79% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobTypeEnum.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobTypeEnum.java index 348ce6e10..813527db2 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobTypeEnum.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/JobTypeEnum.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,9 +16,7 @@ package com.webank.wedatasphere.dss.linkis.node.execution.job; -/** - * Created by peacewong on 2019/11/2. - */ + public enum JobTypeEnum { - CommonJob, EmptyJob,ReadJob, ShareJob + CommonJob, EmptyJob } diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/LinkisJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/LinkisJob.java new file mode 100644 index 000000000..ea6b20dcf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/LinkisJob.java @@ -0,0 +1,43 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.job; + + +import java.util.Map; + + +public abstract class LinkisJob implements Job{ + + + public abstract Map getSource(); + + public abstract void setSource(Map source); + + public abstract JobTypeEnum getJobType(); + public abstract void setJobType(JobTypeEnum jobType); + + public abstract String getSubmitUser(); + + public abstract Map getVariables(); + public abstract void setVariables(Map variables); + + + public abstract Map getConfiguration(); + public abstract void setConfiguration(Map configuration); + + +} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/RuntimeJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/RuntimeJob.java similarity index 83% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/RuntimeJob.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/RuntimeJob.java index 2f803c2ad..3cf3cb4a2 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/RuntimeJob.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/job/RuntimeJob.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,8 +16,6 @@ package com.webank.wedatasphere.dss.linkis.node.execution.job; -/** - * Created by peacewong on 2019/11/2. - */ + public interface RuntimeJob { } diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/listener/LinkisExecutionListener.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/listener/LinkisExecutionListener.java similarity index 86% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/listener/LinkisExecutionListener.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/listener/LinkisExecutionListener.java index 67fa6a786..403bb9a55 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/listener/LinkisExecutionListener.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/listener/LinkisExecutionListener.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,9 +18,7 @@ import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -/** - * Created by peacewong on 2019/11/13. - */ + public interface LinkisExecutionListener { void onStatusChanged(String fromState, String toState, Job job); diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/log/LinkisJobExecutionLog.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/log/LinkisJobExecutionLog.java similarity index 89% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/log/LinkisJobExecutionLog.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/log/LinkisJobExecutionLog.java index cf60ad3da..5aad912a5 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/log/LinkisJobExecutionLog.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/log/LinkisJobExecutionLog.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -17,9 +16,7 @@ package com.webank.wedatasphere.dss.linkis.node.execution.log; -/** - * Created by peacewong on 2019/11/2. - */ + public abstract class LinkisJobExecutionLog { public void info(Object message){ diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/CodeParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/CodeParser.java new file mode 100644 index 000000000..954c95c35 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/CodeParser.java @@ -0,0 +1,221 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.parser; + +import com.google.gson.reflect.TypeToken; +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.entity.BMLResource; +import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; +import com.webank.wedatasphere.dss.linkis.node.execution.job.CommonLinkisJob; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.service.LinkisURLService; +import com.webank.wedatasphere.dss.linkis.node.execution.utils.LinkisJobExecutionUtils; +import com.webank.wedatasphere.linkis.filesystem.WorkspaceClientFactory; +import com.webank.wedatasphere.linkis.filesystem.request.WorkspaceClient; +import com.webank.wedatasphere.linkis.filesystem.response.ScriptFromBMLResponse; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.apache.commons.lang.StringUtils; + + +public class CodeParser implements JobParser { + + private static final Pattern pb = Pattern.compile("((project)|(flow)|(node))://[^\\s\"]+[$\\s]{0,1}", Pattern.CASE_INSENSITIVE); + + @Override + public void parseJob(Job job) throws Exception{ + if (! ( job instanceof CommonLinkisJob) ) { + return ; + } + CommonLinkisJob linkisAppConnJob = (CommonLinkisJob) job; + Map script = LinkisJobExecutionUtils.gson.fromJson(linkisAppConnJob.getCode(), new TypeToken>() {}.getType()); + List jobResourceList = linkisAppConnJob.getJobResourceList(); + BMLResource scriptResource = null; + if (script == null) { + throw new LinkisJobExecutionErrorException(90102,"Script is empty"); + } + String fileName = (String) script.get("script"); + for(BMLResource bmlResource : jobResourceList){ + if(bmlResource.getFileName().equals(fileName)){ + scriptResource = bmlResource; + break; + } + } + if(null == scriptResource) { + throw new LinkisJobExecutionErrorException(90102,"Failed to get script resource"); + } + + Map executionParams = getExecutionParams(scriptResource, linkisAppConnJob); + if (executionParams.get("executionCode") != null) { + String executionCode = (String) executionParams.get("executionCode"); + linkisAppConnJob.getLogObj().info("************************************SUBMIT CODE************************************"); + linkisAppConnJob.getLogObj().info(executionCode); + linkisAppConnJob.getLogObj().info("************************************SUBMIT CODE************************************"); + //parsedCode + linkisAppConnJob.setCode(executionCode); + } + if (executionParams.get("params") != null && executionParams.get("params") instanceof Map) { + if (linkisAppConnJob.getParams() != null) { + linkisAppConnJob.getParams().putAll( (Map)executionParams.get("params")); + } + } + } + + private Map getExecutionParams(BMLResource bmlResource, CommonLinkisJob linkisAppConnJob) { + Map map = new HashMap<>(); + WorkspaceClient client = WorkspaceClientFactory.getClient(linkisAppConnJob.getSubmitUser(), LinkisJobExecutionConfiguration.LINKIS_AUTHOR_USER_TOKEN.getValue(linkisAppConnJob.getJobProps()), + LinkisURLService.Factory.getLinkisURLService().getLinkisURL(linkisAppConnJob)); + ScriptFromBMLResponse response = client.requestOpenScriptFromBML(bmlResource.getResourceId(), bmlResource.getVersion(), bmlResource.getFileName()); + linkisAppConnJob.getLogObj().info("Get execution code from workspace client,bml resource id "+bmlResource.getResourceId()+", version is "+bmlResource.getVersion()); + map.put("executionCode", response.scriptContent()); + map.put("params", response.metadata()); + return map; + } + + private ArrayList getResourceNames(String code){ + ArrayList bmlResourceNames = new ArrayList(); + Matcher mb = pb.matcher(code); + while (mb.find()) { + bmlResourceNames.add(mb.group().trim()); + } + return bmlResourceNames; + } + + + /** + * 1.Find the project file used in the script + * 2.Find the node file used in the script + * 3.Recursively find the flow file used in the script + * 4.Replace file name with prefixed name + * @param resourceNames + * @param linkisAppConnJob + * @return + */ + private ArrayList getResourcesByNames(ArrayList resourceNames, CommonLinkisJob linkisAppConnJob) { + + ArrayList bmlResourceArrayList = new ArrayList<>(); + + String jobName = linkisAppConnJob.getJobName(); + String flowName = linkisAppConnJob.getSource().get("flowName"); + String projectName = linkisAppConnJob.getSource().get("projectName"); + + + List projectResourceList = linkisAppConnJob.getProjectResourceList(); + + + List jobResourceList = linkisAppConnJob.getJobResourceList(); + for (String resourceName : resourceNames) { + String[] resourceNameSplit = resourceName.split("://"); + String prefix = resourceNameSplit[0].toLowerCase(); + String fileName = resourceNameSplit[1]; + BMLResource resource = null; + String afterFileName = fileName; + switch (prefix) { + case "project": + resource = findResource(projectResourceList, fileName); + afterFileName = LinkisJobExecutionConfiguration.PROJECT_PREFIX + "_" + projectName + "_" + fileName; + break; + case "flow": + resource = findFlowResource(linkisAppConnJob, fileName, flowName); + break; + case "node": + resource = findResource(jobResourceList, fileName); + afterFileName = LinkisJobExecutionConfiguration.JOB_PREFIX + "_" + jobName + "_" + fileName; + break; + default: + } + if (null == resource) { + linkisAppConnJob.getLogObj().error("Failed to find the " + prefix + " resource file of " + fileName); + throw new RuntimeException("Failed to find the " + prefix + " resource file of " + fileName); + } + if (!afterFileName.equals(fileName)) { + resource.setFileName(afterFileName); + } + bmlResourceArrayList.add(resource); + } + return bmlResourceArrayList; + } + + + /** + * Recursively find the flow file used in the script + * Recursive exit condition is top-level flow + * + */ + private BMLResource findFlowResource(CommonLinkisJob linkisAppConnJob, String fileName, String flowName) { + + String fullFlowName = ""; + Map> fLowNameAndResources = linkisAppConnJob.getFlowNameAndResources(); + if (fLowNameAndResources == null){ + return null; + } + Optional>> first = fLowNameAndResources.entrySet().stream().filter(fLowNameAndResource -> fLowNameAndResource.getKey().endsWith(flowName + LinkisJobExecutionConfiguration.RESOURCES_NAME)).findFirst(); + + if(first.isPresent()){ + fullFlowName = first.get().getKey(); + BMLResource resource = findResource(first.get().getValue(), fileName); + if (resource != null) { + resource.setFileName(flowName + "_" + fileName); + return resource; + } + } + + String firstFlow = "flow." + flowName + LinkisJobExecutionConfiguration.RESOURCES_NAME; + if (firstFlow.equals(fullFlowName)) { + return null; + } + //getParentFlowName:flow.flows1.test.resources return:flows1 + String parentFlowName = StringUtils.substringAfterLast(StringUtils.substringBefore(fullFlowName, "." + flowName + + LinkisJobExecutionConfiguration.RESOURCES_NAME), "."); + if (StringUtils.isEmpty(parentFlowName)) { + return null; + } + + return findFlowResource(linkisAppConnJob, fileName, parentFlowName); + } + + + private String replaceCodeResourceNames(String code, ArrayList resourceNameList, ArrayList resourceList){ + if(resourceList.size() != resourceNameList.size()){ + throw new RuntimeException("Failed to parsed resource file"); + } + + String[] names = resourceNameList.toArray(new String[]{}); + + String[] afterNames = new String[resourceList.size()]; + for (int i=0 ; i < afterNames.length ; i++){ + afterNames[i] = resourceList.get(i).getFileName(); + } + return StringUtils.replaceEach(code, names, afterNames); + } + + private BMLResource findResource(List resourceArrayList, String fileName){ + if(resourceArrayList != null && !resourceArrayList.isEmpty()) { + for(BMLResource resource : resourceArrayList){ + if(resource.getFileName().equals(fileName)){ + return resource; + } + } + } + return null; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParamsParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParamsParser.java new file mode 100644 index 000000000..44f026f3e --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParamsParser.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.parser; + +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob; +import java.util.Map; + + +public class JobParamsParser implements JobParser { + + @Override + public void parseJob(Job job) { + + if (job instanceof LinkisJob) { + LinkisJob linkisJob = (LinkisJob) job; + linkisJob.getLogObj().info("Start to put variable and configuration"); + //put variable + Map flowVariables = linkisJob.getVariables(); + putParamsMap(job.getParams(), "variable", flowVariables); + // put configuration + Map configuration = linkisJob.getConfiguration(); + putParamsMap(job.getParams(), "configuration", configuration); + linkisJob.getLogObj().info("Finished to put variable and configuration"); + } + } + + private void putParamsMap(Map params, String key, Map value) { + if (null == value) { + return; + } + if (params.get(key) != null) { + ((Map) params.get(key)).putAll(value); + } else { + params.put(key, value); + } + } +} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParser.java similarity index 86% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParser.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParser.java index ce01a5bb9..ef4421e42 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParser.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobParser.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -19,12 +18,9 @@ import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; -/** - * Created by peacewong on 2019/11/2. - */ + public interface JobParser { void parseJob(Job job) throws Exception; - } diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobRuntimeParamsParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobRuntimeParamsParser.java new file mode 100644 index 000000000..8fa5f40bb --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/parser/JobRuntimeParamsParser.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.parser; + +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob; + + +public class JobRuntimeParamsParser implements JobParser{ + + @Override + public void parseJob(Job job) { + if(job instanceof LinkisJob) { + job.getRuntimeParams().put(LinkisJobExecutionConfiguration.LINKIS_SUBMIT_USER,((LinkisJob) job).getSubmitUser()); + } + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/BuildJobAction.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/BuildJobAction.java new file mode 100644 index 000000000..5e901cff2 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/BuildJobAction.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.service; + + +import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; +import com.webank.wedatasphere.linkis.ujes.client.request.JobSubmitAction; + + +public interface BuildJobAction { + + JobExecuteAction getJobAction(Job job) throws LinkisJobExecutionErrorException; + + JobSubmitAction getSubmitAction(Job job) throws LinkisJobExecutionErrorException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/LinkisURLService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/LinkisURLService.java new file mode 100644 index 000000000..77611c6da --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/LinkisURLService.java @@ -0,0 +1,73 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.service; + +import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionWarnException; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.service.impl.LinkisURLServiceImpl; +import com.webank.wedatasphere.linkis.common.utils.ClassUtils; +import java.util.Optional; + + +public interface LinkisURLService { + + /** + * Get Linkis gateway url by job. If multi-Linkis cluster is needed, user can override this method to provide different urls by job. + * @param job this nodeJob + * @return a accessiable Linkis gateway url. + */ + String getLinkisURL(Job job); + + /** + * Get default Linkis gateway url by job. + * @param job this nodeJob + * @return a accessiable Linkis gateway url. + */ + String getDefaultLinkisURL(Job job); + + class Factory { + + private Factory() { + } + + private static LinkisURLService linkisURLService; + + public static LinkisURLService getLinkisURLService() { + if(linkisURLService != null) { + return linkisURLService; + } + synchronized (Factory.class) { + if(linkisURLService == null) { + Optional> clazz = ClassUtils.reflections().getSubTypesOf(LinkisURLService.class).stream().filter(c -> !ClassUtils.isInterfaceOrAbstract(c) && + !LinkisURLServiceImpl.class.isAssignableFrom(c)).findFirst(); + if(clazz.isPresent()) { + try { + linkisURLService = clazz.get().newInstance(); + } catch (Exception e) { + throw new LinkisJobExecutionWarnException(24335, "Get LinkisURLService failed!", e); + } + } else { + linkisURLService = new LinkisURLServiceImpl(); + } + } + } + return linkisURLService; + } + + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/BuildJobActionImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/BuildJobActionImpl.java new file mode 100644 index 000000000..463a81ef1 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/BuildJobActionImpl.java @@ -0,0 +1,191 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.service.impl; + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.exception.LinkisJobExecutionErrorException; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.job.LinkisJob; +import com.webank.wedatasphere.dss.linkis.node.execution.service.BuildJobAction; +import com.webank.wedatasphere.linkis.manager.label.constant.LabelKeyConstant; +import com.webank.wedatasphere.linkis.manager.label.entity.engine.EngineTypeLabel; +import com.webank.wedatasphere.linkis.manager.label.utils.EngineTypeLabelCreator; +import com.webank.wedatasphere.linkis.protocol.utils.TaskUtils; +import com.webank.wedatasphere.linkis.ujes.client.request.JobExecuteAction; +import com.webank.wedatasphere.linkis.ujes.client.request.JobSubmitAction; + +import java.util.HashMap; +import java.util.Map; + +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Map; + +import static com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration.LINKIS_JOB_CREATOR; + + +public class BuildJobActionImpl implements BuildJobAction { + + private Logger logger = LoggerFactory.getLogger(BuildJobActionImpl.class); + private static BuildJobAction buildJobAction = new BuildJobActionImpl(); + + private BuildJobActionImpl() { + + } + + public static BuildJobAction getbuildJobAction() { + return buildJobAction; + } + + private Map prepareYarnLabel(Job job) { + Map labels = new HashMap<>(); + + Object labelStr = job.getRuntimeParams().getOrDefault(LinkisJobExecutionConfiguration.DSS_LABELS_KEY, null); + + if (StringUtils.isNotBlank((String) labelStr)) { + labels.put(LinkisJobExecutionConfiguration.DSS_LABELS_KEY, labelStr); + logger.info("Set Yarn cluster label is: " + labels.toString()); + } else { + labels.put(LinkisJobExecutionConfiguration.DSS_LABELS_KEY, "DEV-DEV"); + logger.warn("Job Label is null. Then set default value"); + } + return labels; + } + + private String parseExecutionCode(Job job) { + String code = job.getCode(); + if (StringUtils.isEmpty(code) || code.equalsIgnoreCase("null")) { + Gson gson = new Gson(); + code = gson.toJson(job.getParams()); + logger.info("The executable code for the job is {}", code); + } + return code; + } + + + @Override + public JobExecuteAction getJobAction(Job job) throws LinkisJobExecutionErrorException { + + enrichParams(job); + + Map labels = prepareYarnLabel(job); + + TaskUtils.addLabelsMap(job.getParams(), labels); + + String code = parseExecutionCode(job); + + JobExecuteAction.Builder builder = JobExecuteAction.builder().setCreator(LINKIS_JOB_CREATOR.getValue(job.getJobProps())) + .addExecuteCode(code) + .setEngineTypeStr(parseAppConnEngineType(job.getEngineType(), job)) + .setRunTypeStr(parseRunType(job.getEngineType(), job.getRunType(), job)) + .setUser(job.getUser()) + .setParams(job.getParams()) + .setRuntimeParams(job.getRuntimeParams()); + if (job instanceof LinkisJob) { + Map source = new HashMap<>(); + source.putAll(((LinkisJob) job).getSource()); + builder = builder.setSource(source); + } + return builder.build(); + } + + @Override + public JobSubmitAction getSubmitAction(Job job) throws LinkisJobExecutionErrorException { + enrichParams(job); + + Map labels = prepareYarnLabel(job); + + TaskUtils.addLabelsMap(job.getParams(), labels); + + String code = parseExecutionCode(job); + + EngineTypeLabel engineTypeLabel = EngineTypeLabelCreator.createEngineTypeLabel(parseAppConnEngineType(job.getEngineType(), job)); + + labels.put(LabelKeyConstant.ENGINE_TYPE_KEY, engineTypeLabel.getStringValue()); + labels.put(LabelKeyConstant.USER_CREATOR_TYPE_KEY, job.getUser() + "-" + LINKIS_JOB_CREATOR.getValue()); + labels.put(LabelKeyConstant.CODE_TYPE_KEY, parseRunType(job.getEngineType(), job.getRunType(), job)); + + + //是否复用引擎，不复用就为空 + if(!isReuseEngine(job.getParams())){ + labels.put("executeOnce", ""); + } + JobSubmitAction.Builder builder = JobSubmitAction.builder().setUser(LINKIS_JOB_CREATOR.getValue(job.getJobProps())) + .addExecuteCode(code) + .setUser(job.getUser()) + .addExecuteUser(job.getUser()) + .setParams(job.getParams()) + .setLabels(labels) + .setRuntimeParams(job.getRuntimeParams()); + if (job instanceof LinkisJob) { + Map source = new HashMap<>(); + source.putAll(((LinkisJob) job).getSource()); + builder = builder.setSource(source); + } + return builder.build(); + } + + /** + * 是否复用引擎，复用返回：true，不复用：false + * @param params + * @return + */ + public boolean isReuseEngine(Map params) { + if (params.get("configuration") != null) { + Map configurationMap = (Map) params.get("configuration"); + if (configurationMap.get("startup") != null) { + Map startupMap = (Map) configurationMap.get("startup"); + if (startupMap.get("ReuseEngine") != null) { + String reuseEngine = (String) startupMap.get("ReuseEngine"); + if (StringUtils.isNotBlank(reuseEngine) && "false".equalsIgnoreCase(reuseEngine.trim())) { + return false; + } + } + } + } + return true; + } + + //TODO + private String parseAppConnEngineType(String engineType, Job job) { + Map props = job.getJobProps(); + if (LinkisJobExecutionConfiguration.isLinkis1_X(props) && engineType.equalsIgnoreCase("appjoint")) { + return LinkisJobExecutionConfiguration.APPCONN; + } + return engineType; + } + + private String parseRunType(String engineType, String runType, Job job) { + Map props = job.getJobProps(); + if (LinkisJobExecutionConfiguration.isLinkis1_X(props) && engineType.equalsIgnoreCase("appjoint")) { + return LinkisJobExecutionConfiguration.APPCONN; + } else if (engineType.toLowerCase().contains("shell")) { + return "shell"; + } + + return runType; + } + + private void enrichParams(Job job) { + job.getRuntimeParams().put("nodeType", job.getRunType()); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/LinkisURLServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/LinkisURLServiceImpl.java new file mode 100644 index 000000000..80ac0469e --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/service/impl/LinkisURLServiceImpl.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.linkis.node.execution.service.impl; + +import com.webank.wedatasphere.dss.linkis.node.execution.conf.LinkisJobExecutionConfiguration; +import com.webank.wedatasphere.dss.linkis.node.execution.job.Job; +import com.webank.wedatasphere.dss.linkis.node.execution.service.LinkisURLService; +import java.util.Map; + + +public class LinkisURLServiceImpl implements LinkisURLService { + + @Override + public String getLinkisURL(Job job) { + Map props = job.getJobProps(); + if(LinkisJobExecutionConfiguration.isLinkis1_X(props)){ + return LinkisJobExecutionConfiguration.LINKIS_URL_1_X.getValue(props); + } else { + return LinkisJobExecutionConfiguration.LINKIS_URL.getValue(props); + } + } + + @Override + public String getDefaultLinkisURL(Job job) { + return LinkisJobExecutionConfiguration.LINKIS_URL_1_X.getValue(job.getJobProps()); + } +} diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisJobExecutionUtils.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisJobExecutionUtils.java similarity index 76% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisJobExecutionUtils.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisJobExecutionUtils.java index 82d56b5ec..6307acfcc 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisJobExecutionUtils.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisJobExecutionUtils.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -26,14 +25,15 @@ import java.util.*; -/** - * Created by peacewong on 2019/9/21. - */ + public class LinkisJobExecutionUtils { + public final static Gson gson = new GsonBuilder().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").create(); + public final static Gson gson1 = new GsonBuilder().disableHtmlEscaping().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").create(); + public static BMLResource getBMLResourceByJson(String json){ @@ -48,12 +48,14 @@ public static ArrayList getResourceListByJson(String json){ } - public static Boolean isCommonAppjointJob(String engineType){ + public static Boolean isCommonAppConnJob(String engineType){ - return LinkisJobExecutionConfiguration.APPJOINT.equalsIgnoreCase(engineType); + return LinkisJobExecutionConfiguration.APPCONN.equalsIgnoreCase(engineType); } + public static final Integer IDX_FOR_LOG_TYPE_ALL = 3;// 0: Error 1: WARN 2:INFO 3: ALL + } diff --git a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisUjesClientUtils.java b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisUjesClientUtils.java similarity index 93% rename from dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisUjesClientUtils.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisUjesClientUtils.java index 61c176d07..fc538b5df 100644 --- a/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisUjesClientUtils.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-linkis-node-execution/src/main/java/com/webank/wedatasphere/dss/linkis/node/execution/utils/LinkisUjesClientUtils.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -27,14 +26,12 @@ import java.util.Map; import java.util.concurrent.TimeUnit; -/** - * Created by peacewong on 2019/9/20. - */ + public class LinkisUjesClientUtils { public static DWSClientConfig getClientConfig(String url, String user, String token, Map jobProps){ DWSClientConfig clientConfig = ((DWSClientConfigBuilder) (DWSClientConfigBuilder.newBuilder() - .addUJESServerUrl(url) + .addServerUrl(url) .connectionTimeout(LinkisJobExecutionConfiguration.LINKIS_CONNECTION_TIMEOUT.getValue(jobProps)) .discoveryEnabled(false).discoveryFrequency(1, TimeUnit.MINUTES) .loadbalancerEnabled(true) diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/pom.xml new file mode 100644 index 000000000..2692dec1b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/pom.xml @@ -0,0 +1,44 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-workflow-common + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.dss + dss-orchestrator-common + ${dss.version} + + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSFlow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSFlow.java new file mode 100644 index 000000000..b3bceb813 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSFlow.java @@ -0,0 +1,222 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.stream.Collectors; + +public class DSSFlow implements Flow { + private Long id; + private String name; + private Boolean state; //0,1代表发布过和未发布过 + private String source; + private String description; + private Date createTime; + private String creator; + private Boolean isRootFlow; + private Integer rank; + private Long projectID; + private String linkedAppConnNames; + private String dssLabels; + + /** + * 0disable 1 enable 0表示工作流从来没存过，发布的时候忽略 + */ + + private Boolean hasSaved; + private String uses; + private List children; + private String flowType; + private String resourceId; + + private String bmlVersion; + + public String getFlowJson() { + return flowJson; + } + + public void setFlowJson(String flowJson) { + this.flowJson = flowJson; + } + + private String flowJson; + + public String getResourceId() { + return resourceId; + } + + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + public String getBmlVersion() { + return bmlVersion; + } + + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + public Integer getRank() { + return rank; + } + + public void setRank(Integer rank) { + this.rank = rank; + } + + @Override + public Long getId() { + return id; + } + + @Override + public void setId(Long id) { + this.id = id; + } + + @Override + public String getName() { + return name; + } + + @Override + public void setName(String name) { + this.name = name; + } + + + @Override + public String getDescription() { + return description; + } + + @Override + public void setDescription(String description) { + this.description = description; + } + + @Override + public List getChildren() { + return children; + } + + public void addChildren(DSSFlow children) { + if (this.children == null){ + this.children = new ArrayList<>(); + } + this.children.add(children); + } + + @Override + public void setChildren(List children) { + this.children = children.stream().map(f -> (DSSFlow) f).collect(Collectors.toList()); + } + + @Override + public String getFlowType() { + return flowType; + } + + @Override + public void setFlowType(String flowType) { + this.flowType = flowType; + } + + @Override + public Boolean getRootFlow() { + return isRootFlow; + } + + public void setRootFlow(Boolean rootFlow) { + isRootFlow = rootFlow; + } + + public Boolean getState() { + return state; + } + + public void setState(Boolean state) { + this.state = state; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public String getCreator() { + return creator; + } + + public void setCreator(String creator) { + this.creator = creator; + } + + public Long getProjectID() { + return projectID; + } + + public void setProjectID(Long projectID) { + this.projectID = projectID; + } + + public Boolean getHasSaved() { + return hasSaved; + } + + public void setHasSaved(Boolean hasSaved) { + this.hasSaved = hasSaved; + } + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public String getLinkedAppConnNames() { + return linkedAppConnNames; + } + + public void setLinkedAppConnNames(String linkedAppConnNames) { + this.linkedAppConnNames = linkedAppConnNames; + } + + public String getDssLabels() { + return dssLabels; + } + + public void setDssLabels(String dssLabels) { + this.dssLabels = dssLabels; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSFlowRelation.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSFlowRelation.java new file mode 100644 index 000000000..2d5a35ac5 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSFlowRelation.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + +public class DSSFlowRelation { + private Long id; + private Long flowID; + private Long parentFlowID; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getFlowID() { + return flowID; + } + + public void setFlowID(Long flowID) { + this.flowID = flowID; + } + + public Long getParentFlowID() { + return parentFlowID; + } + + public void setParentFlowID(Long parentFlowID) { + this.parentFlowID = parentFlowID; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSJsonFlow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSJsonFlow.java new file mode 100644 index 000000000..4635338c7 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSJsonFlow.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + +import java.util.List; +import java.util.stream.Collectors; + +public class DSSJsonFlow extends DSSFlow { + private String json; + + private List children; + + public String getJson() { + return json; + } + + public void setJson(String json) { + this.json = json; + } + + @Override + public void setChildren(List children) { + this.children = children.stream().map(f ->(DSSJsonFlow)f).collect(Collectors.toList()); + } + + @Override + public List getChildren() { + return children; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSPersonalFlow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSPersonalFlow.java new file mode 100644 index 000000000..6447b3795 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DSSPersonalFlow.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + + +public class DSSPersonalFlow extends DSSJsonFlow { + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DWSFlowPublishHistory.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DWSFlowPublishHistory.java new file mode 100644 index 000000000..62a88ecd5 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DWSFlowPublishHistory.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + + +public class DWSFlowPublishHistory { +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DWSFlowVersion.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DWSFlowVersion.java new file mode 100644 index 000000000..80c665e00 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/DWSFlowVersion.java @@ -0,0 +1,191 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + +import java.util.Date; + + +public class DWSFlowVersion implements FlowVersion, Comparable { + private Long id; + private Long flowID; + private String source; + private String jsonPath; + private String comment; + private Date updateTime; + private String version; + private String bmlVersion; + private DWSFlowPublishHistory publishHistory; + private String json; + private String updator; + private Boolean isNotPublished; //true 未发过版，false已经过版 + private Long projectVersionID; + private Long oldFlowID; + /** + * 工作流是否被锁 + */ + private Boolean isFlowEditLockExist; + + public Boolean getFlowEditLockExist() { + return isFlowEditLockExist; + } + + public void setFlowEditLockExist(Boolean flowEditLockExist) { + isFlowEditLockExist = flowEditLockExist; + } + + /** + * 工作流编辑锁，打开自动获取，超时自动释放 + */ + private String flowEditLock; + + public String getFlowEditLock() { + return flowEditLock; + } + + public void setFlowEditLock(String flowEditLock) { + this.flowEditLock = flowEditLock; + } + + public Long getOldFlowID() { + return oldFlowID; + } + + public void setOldFlowID(Long oldFlowID) { + this.oldFlowID = oldFlowID; + } + + public Long getProjectVersionID() { + return projectVersionID; + } + + public void setProjectVersionID(Long projectVersionID) { + this.projectVersionID = projectVersionID; + } + + public Boolean getNotPublished() { + return isNotPublished; + } + + public void setNotPublished(Boolean notPublished) { + isNotPublished = notPublished; + } + + public String getUpdator() { + return updator; + } + + public void setUpdator(String updator) { + this.updator = updator; + } + + public String getJson() { + return json; + } + + public void setJson(String json) { + this.json = json; + } + + @Override + public String getVersion() { + return version; + } + + @Override + public void setVersion(String version) { + this.version = version; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getFlowID() { + return flowID; + } + + public void setFlowID(Long flowID) { + this.flowID = flowID; + } + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getJsonPath() { + return jsonPath; + } + + public void setJsonPath(String jsonPath) { + this.jsonPath = jsonPath; + } + + @Override + public String getComment() { + return comment; + } + + @Override + public void setComment(String comment) { + this.comment = comment; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } + + public DWSFlowPublishHistory getPublishHistory() { + return publishHistory; + } + + public void setPublishHistory(DWSFlowPublishHistory publishHistory) { + this.publishHistory = publishHistory; + } + + public String getBmlVersion() { + return bmlVersion; + } + + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + @Override + public int compareTo(DWSFlowVersion o) { + Integer v1 = Integer.valueOf(this.version.substring(1, version.length())); + Integer v2 = Integer.valueOf(o.version.substring(1, o.version.length())); + if (v1.equals(v2) && this.bmlVersion != null && o.bmlVersion != null) { + Integer bV1 = Integer.valueOf(this.bmlVersion.substring(1, bmlVersion.length())); + Integer bV2 = Integer.valueOf(o.bmlVersion.substring(1, bmlVersion.length())); + return bV2 - bV1; + } else { + return v2 - v1; + } + } +} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FileFlowVersion.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FileFlowVersion.java similarity index 79% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FileFlowVersion.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FileFlowVersion.java index a249d030a..ccd558d0c 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FileFlowVersion.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FileFlowVersion.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,10 +14,10 @@ * */ -package com.webank.wedatasphere.dss.common.entity.flow; +package com.webank.wedatasphere.dss.workflow.common.entity; -/** - * Created by enjoyyin on 2019/5/14. +/* + FileFlowVersion 接口 */ public interface FileFlowVersion extends FlowVersion { Object getFlowFile(); diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/Flow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/Flow.java new file mode 100644 index 000000000..ef8dc85c6 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/Flow.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.entity; + +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestration; +import java.util.List; + +public interface Flow extends DSSOrchestration { + + void setId(Long id); + + void setName(String name); + + String getFlowType(); + + void setFlowType(String flowType); + + void setDescription(String description); + + Boolean getRootFlow(); + + List getChildren(); + + void setChildren(List children); + +} diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FlowVersion.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FlowVersion.java similarity index 79% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FlowVersion.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FlowVersion.java index 747bab71c..044f6d87d 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FlowVersion.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FlowVersion.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,11 +14,10 @@ * */ -package com.webank.wedatasphere.dss.common.entity.flow; - -/** - * Created by enjoyyin on 2019/5/14. - */ +package com.webank.wedatasphere.dss.workflow.common.entity; +/* + 顶级FlowVersion接口 +*/ public interface FlowVersion { String getVersion(); diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FlowVersionForNodes.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FlowVersionForNodes.java similarity index 82% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FlowVersionForNodes.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FlowVersionForNodes.java index f61c596ea..8001e9a4c 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/FlowVersionForNodes.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/FlowVersionForNodes.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,14 +14,14 @@ * */ -package com.webank.wedatasphere.dss.common.entity.flow; +package com.webank.wedatasphere.dss.workflow.common.entity; import com.webank.wedatasphere.dss.common.entity.node.Node; import java.util.List; -/** - * Created by enjoyyin on 2019/5/14. +/* + FlowVersionForNodes 接口 */ public interface FlowVersionForNodes { List getNodes(); diff --git a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/JSONFlowVersion.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/JSONFlowVersion.java similarity index 78% rename from dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/JSONFlowVersion.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/JSONFlowVersion.java index 22be80d81..0ed6e4c21 100644 --- a/dss-common/src/main/java/com/webank/wedatasphere/dss/common/entity/flow/JSONFlowVersion.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/entity/JSONFlowVersion.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,11 +14,11 @@ * */ -package com.webank.wedatasphere.dss.common.entity.flow; +package com.webank.wedatasphere.dss.workflow.common.entity; -/** - * Created by enjoyyin on 2019/5/14. - */ +/* + JSONFlowVersion 接口 +*/ public interface JSONFlowVersion extends FlowVersion { Object getFlowJSON(); diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/parser/NodeParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/parser/NodeParser.java new file mode 100644 index 000000000..53721e040 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/parser/NodeParser.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.parser; + + +import com.webank.wedatasphere.dss.common.entity.Resource; + +import java.io.IOException; +import java.util.List; +import java.util.Map; + + +public interface NodeParser { + String updateNodeResource(String nodeJson, List resources) throws IOException; + String updateNodeJobContent(String nodeJson, Map content) throws IOException; + String updateSubFlowID(String nodeJson, long subflowId) throws IOException; + String getNodeValue(String key, String nodeJson) throws IOException; + List getNodeResource(String nodeJson); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/parser/WorkFlowParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/parser/WorkFlowParser.java new file mode 100644 index 000000000..5cefbbd4c --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/parser/WorkFlowParser.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.parser; + + +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; + +import java.io.IOException; +import java.util.List; + + +public interface WorkFlowParser { + List getWorkFlowResources(String workFlowJson); + List getWorkFlowNodes(String workFlowJson); + List getWorkFlowNodesJson(String workFlowJson); + String updateFlowJsonWithKey(String workFlowJson, String key, Object value) throws IOException; + String getValueWithKey(String workFlowJson, String key) throws IOException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestCopyWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestCopyWorkflow.java new file mode 100644 index 000000000..daec88299 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestCopyWorkflow.java @@ -0,0 +1,112 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + + +public class RequestCopyWorkflow { + private String userName; + private String workspaceName; + private Long rootFlowId; + private String contextIdStr; + private String projectName; + private Long orcVersionId; + private String version; + private String description; + + public RequestCopyWorkflow(String userName, + String workspaceName, + Long rootFlowId, + String contextIdStr, + String projectName, + Long orcVersionId, + String version, + String description) { + + this.userName = userName; + this.workspaceName = workspaceName; + this.rootFlowId = rootFlowId; + this.contextIdStr = contextIdStr; + this.projectName = projectName; + this.orcVersionId = orcVersionId; + this.version = version; + this.description = description; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public Long getRootFlowId() { + return rootFlowId; + } + + public void setRootFlowId(Long rootFlowId) { + this.rootFlowId = rootFlowId; + } + + public String getContextIdStr() { + return contextIdStr; + } + + public void setContextIdStr(String contextIdStr) { + this.contextIdStr = contextIdStr; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public Long getOrcVersionId() { + return orcVersionId; + } + + public void setOrcVersionId(Long orcVersionId) { + this.orcVersionId = orcVersionId; + } + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestCreateWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestCreateWorkflow.java new file mode 100644 index 000000000..dd8cc9026 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestCreateWorkflow.java @@ -0,0 +1,115 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; + + +public class RequestCreateWorkflow { + + private String userName; + private String workflowName; + private String contextIDStr; + private String description; + private Long parentFlowID; + private String uses; + private List linkedAppConnNames; + private List dssLabels; + + public RequestCreateWorkflow(String userName, + String workflowName, + String contextIDStr, + String description, + Long parentFlowID, + String uses, + List linkedAppConnNames, + List dssLabels) { + this.userName = userName; + this.workflowName = workflowName; + this.contextIDStr = contextIDStr; + this.description = description; + this.parentFlowID = parentFlowID; + this.uses = uses; + this.linkedAppConnNames = linkedAppConnNames; + this.dssLabels =dssLabels; + } + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getWorkflowName() { + return workflowName; + } + + public void setWorkflowName(String workflowName) { + this.workflowName = workflowName; + } + + public String getContextIDStr() { + return contextIDStr; + } + + public void setContextIDStr(String contextIDStr) { + this.contextIDStr = contextIDStr; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Long getParentFlowID() { + return parentFlowID; + } + + public void setParentFlowID(Long parentFlowID) { + this.parentFlowID = parentFlowID; + } + + public String getUses() { + return uses; + } + + public void setUses(String uses) { + this.uses = uses; + } + + public List getLinkedAppConnNames() { + return linkedAppConnNames; + } + + public void setLinkedAppConnNames(List linkedAppConnNames) { + this.linkedAppConnNames = linkedAppConnNames; + } + public List getDssLabels() { + return dssLabels; + } + + public void setDssLabels(List dssLabels) { + this.dssLabels = dssLabels; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestImportWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestImportWorkflow.java new file mode 100644 index 000000000..d7e8f4267 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/RequestImportWorkflow.java @@ -0,0 +1,140 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; + + +public class RequestImportWorkflow { + private String userName; + private String resourceId; + private String bmlVersion; + private Long projectId; + private String projectName; + private IOEnv sourceEnv; + private String orcVersion; + private String workspaceName; + private String workspaceStr; + private String contextId; + + + public String getContextId() { + return contextId; + } + + public void setContextId(String contextId) { + this.contextId = contextId; + } + + + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + public String getResourceId() { + return resourceId; + } + + public void setResourceId(String resourceId) { + this.resourceId = resourceId; + } + + public String getBmlVersion() { + return bmlVersion; + } + + public void setBmlVersion(String bmlVersion) { + this.bmlVersion = bmlVersion; + } + + public Long getProjectId() { + return projectId; + } + + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public IOEnv getSourceEnv() { + return sourceEnv; + } + + public void setSourceEnv(IOEnv sourceEnv) { + this.sourceEnv = sourceEnv; + } + + public String getOrcVersion() { + return orcVersion; + } + + public void setOrcVersion(String orcVersion) { + this.orcVersion = orcVersion; + } + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public String getWorkspaceStr() { + return workspaceStr; + } + + public void setWorkspaceStr(String workspaceStr) { + this.workspaceStr = workspaceStr; + } + + public RequestImportWorkflow(String userName, + String resourceId, + String bmlVersion, + Long projectId, + String projectName, + IOEnv sourceEnv, + String orcVersion, + String workspaceName, + String workspaceStr, + String contextId) { + this.userName = userName; + this.resourceId = resourceId; + this.bmlVersion = bmlVersion; + this.projectId = projectId; + this.projectName = projectName; + this.sourceEnv = sourceEnv; + this.orcVersion = orcVersion; + this.workspaceName = workspaceName; + this.workspaceStr = workspaceStr; + this.contextId = contextId; + } + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseCopyWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseCopyWorkflow.java new file mode 100644 index 000000000..07b59a607 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseCopyWorkflow.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; + + +public class ResponseCopyWorkflow { + + private DSSFlow dssFlow; + + public ResponseCopyWorkflow(DSSFlow dssFlow) { + this.dssFlow = dssFlow; + } + + public DSSFlow getDssFlow() { + return dssFlow; + } + + public void setDssFlow(DSSFlow dssFlow) { + this.dssFlow = dssFlow; + } + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseCreateWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseCreateWorkflow.java new file mode 100644 index 000000000..9f4443cd3 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseCreateWorkflow.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; + + +public class ResponseCreateWorkflow { + + public DSSFlow getDssFlow() { + return dssFlow; + } + + public void setDssFlow(DSSFlow dssFlow) { + this.dssFlow = dssFlow; + } + + private DSSFlow dssFlow; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseDeleteWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseDeleteWorkflow.java new file mode 100644 index 000000000..ad8273435 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseDeleteWorkflow.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; + + +public class ResponseDeleteWorkflow { + private JobStatus jobStatus; + + public ResponseDeleteWorkflow(JobStatus jobStatus) { + this.jobStatus = jobStatus; + } + + public JobStatus getJobStatus() { + return jobStatus; + } + + public void setJobStatus(JobStatus jobStatus) { + this.jobStatus = jobStatus; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseImportWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseImportWorkflow.java new file mode 100644 index 000000000..3cb36ed8f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseImportWorkflow.java @@ -0,0 +1,51 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; + +import java.util.List; + + +public class ResponseImportWorkflow { + + + private JobStatus status; + + private List workflowIds; + + public ResponseImportWorkflow(JobStatus status, List workflowIds) { + this.status = status; + this.workflowIds = workflowIds; + } + + public JobStatus getStatus() { + return status; + } + + public void setStatus(JobStatus status) { + this.status = status; + } + + public List getWorkflowIds() { + return workflowIds; + } + + public void setWorkflowIds(List workflowIds) { + this.workflowIds = workflowIds; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseQueryWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseQueryWorkflow.java new file mode 100644 index 000000000..4ea22adad --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseQueryWorkflow.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; + + +public class ResponseQueryWorkflow { + + private DSSFlow dssFlow; + + public ResponseQueryWorkflow(DSSFlow dssFlow) { + this.dssFlow = dssFlow; + } + + + + public DSSFlow getDssFlow() { + return dssFlow; + } + + public void setDssFlow(DSSFlow dssFlow) { + this.dssFlow = dssFlow; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseUpdateWorkflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseUpdateWorkflow.java new file mode 100644 index 000000000..3a87286e4 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-common/src/main/java/com/webank/wedatasphere/dss/workflow/common/protocol/ResponseUpdateWorkflow.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.common.protocol; + +import com.webank.wedatasphere.dss.common.protocol.JobStatus; + + +public class ResponseUpdateWorkflow { + private JobStatus jobStatus; + + public ResponseUpdateWorkflow(JobStatus jobStatus) { + this.jobStatus = jobStatus; + } + + public JobStatus getJobStatus() { + return jobStatus; + } + + public void setJobStatus(JobStatus jobStatus) { + this.jobStatus = jobStatus; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/pom.xml new file mode 100644 index 000000000..01d1b54a5 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/pom.xml @@ -0,0 +1,43 @@ + + + + + + dss-workflow + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-workflow-conversion-standard + + + + com.webank.wedatasphere.dss + dss-workflow-sdk + ${dss.version} + + + com.webank.wedatasphere.dss + dss-orchestrator-conversion-standard + ${dss.version} + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/WorkflowConversionIntegrationStandard.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/WorkflowConversionIntegrationStandard.java new file mode 100644 index 000000000..d830550e0 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/WorkflowConversionIntegrationStandard.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.AbstractConversionIntegrationStandard; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.DSSToRelConversionService; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.RelToOrchestratorConversionService; +import com.webank.wedatasphere.dss.workflow.conversion.service.WorkflowToRelConversionService; + + +public class WorkflowConversionIntegrationStandard extends AbstractConversionIntegrationStandard { + + @Override + protected DSSToRelConversionService createDSSToRelConversionService() { + return new WorkflowToRelConversionService(); + } + + @Override + protected RelToOrchestratorConversionService createRelToDSSConversionService() { + return null; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/ConvertedRel.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/ConvertedRel.java new file mode 100644 index 000000000..8e6d4c8ad --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/ConvertedRel.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.entity; + + +public interface ConvertedRel extends PreConversionRel { + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/PreConversionRel.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/PreConversionRel.java new file mode 100644 index 000000000..374431efc --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/PreConversionRel.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.entity; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.DSSToRelConversionRequestRef; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import java.util.List; + + +public interface PreConversionRel { + + List getWorkflows(); + + DSSToRelConversionRequestRef getDSSToRelConversionRequestRef(); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/PreConversionRelImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/PreConversionRelImpl.java new file mode 100644 index 000000000..ea88e3519 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/entity/PreConversionRelImpl.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.entity; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.DSSToRelConversionRequestRef; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import java.util.List; + + +public class PreConversionRelImpl implements PreConversionRel { + + private List workflows; + private DSSToRelConversionRequestRef dssToRelConversionRequestRef; + + public void setWorkflows(List workflows) { + this.workflows = workflows; + } + + public void setDSSToRelConversionRequestRef( + DSSToRelConversionRequestRef dssToRelConversionRequestRef) { + this.dssToRelConversionRequestRef = dssToRelConversionRequestRef; + } + + @Override + public List getWorkflows() { + return workflows; + } + + @Override + public DSSToRelConversionRequestRef getDSSToRelConversionRequestRef() { + return dssToRelConversionRequestRef; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelConversionOperation.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelConversionOperation.java new file mode 100644 index 000000000..c72ebbe65 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelConversionOperation.java @@ -0,0 +1,96 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.operation; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.operation.DSSToRelConversionOperation; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.DSSToRelConversionRequestRef; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.ProjectToRelConversionRequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.CommonResponseRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.utils.AppStandardClassUtils; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRelImpl; +import com.webank.wedatasphere.dss.workflow.core.WorkflowFactory; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.json2flow.AbstractJsonToFlowParser; +import com.webank.wedatasphere.dss.workflow.core.json2flow.JsonToFlowParser; +import com.webank.wedatasphere.dss.workflow.core.json2flow.parser.WorkflowParser; +import java.util.Comparator; +import java.util.List; +import java.util.stream.Collectors; + + +public class WorkflowToRelConversionOperation extends DSSToRelConversionOperation { + + private List workflowToRelConverters; + private WorkflowToRelSynchronizer workflowToRelSynchronizer; + + public WorkflowToRelConversionOperation(){ + } + + @Override + public void init() { + String appConnName = getConversionService().getAppStandard().getAppConnName(); + workflowToRelConverters = AppStandardClassUtils.getInstance(appConnName).getInstances(WorkflowToRelConverter.class).stream().sorted(Comparator.comparingInt(WorkflowToRelConverter::getOrder)) + .collect(Collectors.toList()); + workflowToRelSynchronizer = AppStandardClassUtils.getInstance(appConnName).getInstanceOrWarn(WorkflowToRelSynchronizer.class); + workflowToRelSynchronizer.setAppInstance(getConversionService().getAppInstance()); + workflowToRelSynchronizer.setSSORequestService(getConversionService().getSSORequestService()); + JsonToFlowParser parser = WorkflowFactory.INSTANCE.getJsonToFlowParser(); + if(parser instanceof AbstractJsonToFlowParser) { + String packageName = WorkflowParser.class.getPackage().getName(); + List workflowParsers = AppStandardClassUtils.getInstance(appConnName).getInstances(WorkflowParser.class).stream() + .filter(p -> !p.getClass().getName().startsWith(packageName)).collect(Collectors.toList()); + ((AbstractJsonToFlowParser) parser).addWorkflowParsers(workflowParsers); + } + } + + @Override + public ResponseRef convert(DSSToRelConversionRequestRef ref) { + List workflows; + if(ref instanceof ProjectToRelConversionRequestRef) { + ProjectToRelConversionRequestRef projectRef = (ProjectToRelConversionRequestRef) ref; + workflows = projectRef.getDSSOrcList().stream().map(flow -> WorkflowFactory.INSTANCE.getJsonToFlowParser().parse((DSSFlow) flow)) + .collect(Collectors.toList()); + } else { + return CommonResponseRef.error("Not support ref " + ref.getClass().getSimpleName()); + } + ConvertedRel convertedRel = tryConvert(workflows, ref); + trySync(convertedRel); + return CommonResponseRef.success("All workflow convert succeed!"); + } + + protected ConvertedRel tryConvert(List workflows, DSSToRelConversionRequestRef ref){ + PreConversionRelImpl rel = new PreConversionRelImpl(); + rel.setWorkflows(workflows); + rel.setDSSToRelConversionRequestRef(ref); + ConvertedRel convertedRel = null; + for (WorkflowToRelConverter workflowToRelConverter: workflowToRelConverters) { + if(convertedRel == null) { + convertedRel = workflowToRelConverter.convertToRel(rel); + } else { + convertedRel = workflowToRelConverter.convertToRel(convertedRel); + } + } + return convertedRel; + } + + protected void trySync(ConvertedRel convertedRel) { + workflowToRelSynchronizer.syncToRel(convertedRel); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelConverter.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelConverter.java new file mode 100644 index 000000000..7ef3f916f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelConverter.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.operation; + +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; +import com.webank.wedatasphere.dss.workflow.conversion.entity.PreConversionRel; +import com.webank.wedatasphere.dss.workflow.core.order.Order; + + +public interface WorkflowToRelConverter extends Order { + + /** + * Try to convert DSS workflow to external app ref. + * Warn: The implementation should resolve the loop of children workflow's by yourself. + * @param rel The DSS workflow wait to convert + * @return Converted Rel + */ + ConvertedRel convertToRel(PreConversionRel rel); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelSynchronizer.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelSynchronizer.java new file mode 100644 index 000000000..25ddb48cf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/operation/WorkflowToRelSynchronizer.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.operation; + +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.workflow.conversion.entity.ConvertedRel; + + +public interface WorkflowToRelSynchronizer { + + void setAppInstance(AppInstance appInstance); + + void syncToRel(ConvertedRel convertedRel); + + void setSSORequestService(SSORequestService ssoRequestService); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/service/WorkflowToRelConversionService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/service/WorkflowToRelConversionService.java new file mode 100644 index 000000000..0d9046d27 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-conversion-standard/src/main/java/com/webank/wedatasphere/dss/workflow/conversion/service/WorkflowToRelConversionService.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conversion.service; + +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.AbstractConversionService; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.service.DSSToRelConversionService; +import com.webank.wedatasphere.dss.workflow.conversion.operation.WorkflowToRelConversionOperation; + + +public class WorkflowToRelConversionService extends AbstractConversionService + implements DSSToRelConversionService { + + @Override + public WorkflowToRelConversionOperation getDSSToRelConversionOperation() { + return getOrCreate(WorkflowToRelConversionOperation::new, WorkflowToRelConversionOperation.class); + } + + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/pom.xml new file mode 100644 index 000000000..04bf9c565 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/pom.xml @@ -0,0 +1,53 @@ + + + + + + dss-workflow + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-workflow-sdk + + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + + + com.webank.wedatasphere.dss + dss-workflow-common + ${dss.version} + + + org.springframework + spring-context + ${spring.version} + provided + + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/WorkflowFactory.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/WorkflowFactory.java new file mode 100644 index 000000000..09ede336a --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/WorkflowFactory.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core; + +import com.webank.wedatasphere.dss.workflow.core.builder.WorkflowBuilder; +import com.webank.wedatasphere.dss.workflow.core.flow2json.FlowToJsonParser; +import com.webank.wedatasphere.dss.workflow.core.json2flow.JsonToFlowParser; + + +public interface WorkflowFactory { + + WorkflowBuilder createWorkflowBuilder(); + + JsonToFlowParser getJsonToFlowParser(); + + FlowToJsonParser getFlowToJsonParser(); + + WorkflowFactory INSTANCE = WorkflowFactoryImpl.createInstance(); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/WorkflowFactoryImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/WorkflowFactoryImpl.java new file mode 100644 index 000000000..d67f0a457 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/WorkflowFactoryImpl.java @@ -0,0 +1,76 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.workflow.core.builder.WorkflowBuilder; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowImpl; +import com.webank.wedatasphere.dss.workflow.core.flow2json.FlowToJsonParser; +import com.webank.wedatasphere.dss.workflow.core.json2flow.AbstractJsonToFlowParser; +import com.webank.wedatasphere.dss.workflow.core.json2flow.JsonToFlowParser; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + + +public class WorkflowFactoryImpl implements WorkflowFactory { + + private static final Logger LOGGER = LoggerFactory.getLogger(WorkflowFactoryImpl.class); + + private JsonToFlowParser jsonToFlowParser; + + @Override + public WorkflowBuilder createWorkflowBuilder() { + return null; + } + + @Override + public JsonToFlowParser getJsonToFlowParser() { + if(jsonToFlowParser != null) { + return jsonToFlowParser; + } + synchronized (this) { + if(jsonToFlowParser == null) { + try { + jsonToFlowParser = ClassUtils.getInstance(JsonToFlowParser.class); + } catch (DSSErrorException e) { + jsonToFlowParser = new AbstractJsonToFlowParser() { + @Override + protected Workflow createWorkflow() { + return new WorkflowImpl(); + } + }; + } + jsonToFlowParser.init(); + LOGGER.info("JsonToFlowParser is {}.", jsonToFlowParser.getClass().getSimpleName()); + } + } + return jsonToFlowParser; + } + + @Override + public FlowToJsonParser getFlowToJsonParser() { + return null; + } + + static WorkflowFactory createInstance() { + WorkflowFactory workflowFactory = ClassUtils.getInstanceOrDefault(WorkflowFactory.class, new WorkflowFactoryImpl()); + LOGGER.info("WorkflowFactory is {}.", workflowFactory.getClass().getSimpleName()); + return workflowFactory; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/builder/WorkflowBuilder.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/builder/WorkflowBuilder.java new file mode 100644 index 000000000..3ed453b46 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/builder/WorkflowBuilder.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.builder; + +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import java.util.Map; + +/** + * More conveniently to build a workflow. + */ +public interface WorkflowBuilder { + + T create(String name, String description); + + T addWorkflowNode(WorkflowNode node); + + T addWorkflowNode(String parentNode, WorkflowNode node); + + T addChild(Workflow child); + + T addChild(String parentNode, Workflow child); + + T addFlowResource(Resource resource); + + T addFlowProperties(Map properties); + + Workflow build(); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/constant/WorkflowConstant.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/constant/WorkflowConstant.java new file mode 100644 index 000000000..a25e3d074 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/constant/WorkflowConstant.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.constant; + + +public class WorkflowConstant { + public final static String PROXY_USER = "user.to.proxy"; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/Workflow.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/Workflow.java new file mode 100644 index 000000000..b1f0d8f98 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/Workflow.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.workflow.common.entity.Flow; +import java.util.List; +import java.util.Map; + + +public interface Workflow extends Flow { + + Workflow getParentWorkflow(); + + List getWorkflowNodes(); + + List getWorkflowNodeEdges(); + + List getFlowResources(); + + List> getFlowProperties(); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowImpl.java new file mode 100644 index 000000000..3eb70cd97 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowImpl.java @@ -0,0 +1,166 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSJsonFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.Flow; + +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +public class WorkflowImpl implements Workflow { + + private Long id; + private String name; + private String description; + private String type; + private List children; + private Boolean isRootFlow; + private String userProxy; + private List workflowNodes; + private List workflowNodeEdges; + private List flowResources; + private List> flowProperties; + private DSSJsonFlow jsonFlow; + private Workflow parentWorkflow; + + public DSSJsonFlow getJsonFlow() { + return jsonFlow; + } + + public void setJsonFlow(DSSJsonFlow jsonFlow) { + this.jsonFlow = jsonFlow; + } + + @Override + public Workflow getParentWorkflow() { + return parentWorkflow; + } + + public void setParentWorkflow(WorkflowImpl parentWorkflow) { + this.parentWorkflow = parentWorkflow; + } + + public String getUserProxy() { + return userProxy; + } + + public void setUserProxy(String userProxy) { + this.userProxy = userProxy; + } + + @Override + public List getWorkflowNodes() { + return workflowNodes; + } + + public void setWorkflowNodes(List workflowNodes) { + this.workflowNodes = workflowNodes; + } + + @Override + public List getWorkflowNodeEdges() { + return workflowNodeEdges; + } + + public void setWorkflowNodeEdges(List workflowNodeEdges) { + this.workflowNodeEdges = workflowNodeEdges; + } + + @Override + public List getFlowResources() { + return flowResources; + } + + public void setFlowResources(List flowResources) { + this.flowResources = flowResources; + } + + @Override + public List> getFlowProperties() { + return flowProperties; + } + + public void setFlowProperties(List> flowProperties) { + this.flowProperties = flowProperties; + } + + @Override + public Long getId() { + return this.id; + } + + @Override + public void setId(Long id) { + this.id = id; + } + + @Override + public String getName() { + return this.name; + } + + @Override + public void setName(String name) { + this.name = name; + } + + @Override + public String getFlowType() { + return this.type; + } + + @Override + public void setFlowType(String flowType) { + this.type = flowType; + + } + + @Override + public String getDescription() { + return this.description; + } + + @Override + public void setDescription(String description) { + this.description = description; + } + + @Override + public Boolean getRootFlow() { + return this.isRootFlow; + } + + public void setRootFlow(Boolean rootFlow) { + isRootFlow = rootFlow; + } + + @Override + public List getChildren() { + return this.children; + } + + @Override + public void setChildren(List children) { + if(null!=children) { + this.children = children.stream().map(f -> (WorkflowImpl) f).collect(Collectors.toList()); + } + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNode.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNode.java new file mode 100644 index 000000000..a4908dedf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNode.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + + +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.entity.node.Node; + +public interface WorkflowNode extends Node { + + DSSNode getDSSNode(); + + void setDSSNode(DSSNode dwsNode); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeEdge.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeEdge.java new file mode 100644 index 000000000..5669014f1 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeEdge.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + +import com.webank.wedatasphere.dss.common.entity.node.DSSEdge; + +public interface WorkflowNodeEdge { + DSSEdge getDSSEdge(); + + void setDSSEdge(DSSEdge dssEdge); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeEdgeImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeEdgeImpl.java new file mode 100644 index 000000000..40f726eaf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeEdgeImpl.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + +import com.webank.wedatasphere.dss.common.entity.node.DSSEdge; + + +public class WorkflowNodeEdgeImpl implements WorkflowNodeEdge { + private DSSEdge dssEdge; + + @Override + public DSSEdge getDSSEdge() { + return dssEdge; + } + + @Override + public void setDSSEdge(DSSEdge dssEdge) { + this.dssEdge = dssEdge; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeImpl.java new file mode 100644 index 000000000..5ae20ece0 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowNodeImpl.java @@ -0,0 +1,87 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; + +import java.util.List; + + +public class WorkflowNodeImpl implements WorkflowNode { + + private DSSNode dssNode; + + @Override + public DSSNode getDSSNode() { + return this.dssNode; + } + + @Override + public void setDSSNode(DSSNode dssNode) { + this.dssNode = dssNode; + } + + @Override + public String getId() { + return dssNode.getId(); + } + + @Override + public void setId(String id) { + dssNode.setId(id); + } + + @Override + public String getNodeType() { + return dssNode.getNodeType(); + } + + @Override + public void setNodeType(String nodeType) { + dssNode.setNodeType(nodeType); + } + + @Override + public String getName() { + return dssNode.getName(); + } + + @Override + public void setName(String name) { + dssNode.setName(name); + } + + @Override + public void addDependency(String nodeName) { + dssNode.addDependency(nodeName); + } + + @Override + public void setDependency(List dependency) { + dssNode.setDependency(dependency); + } + + @Override + public void removeDependency(String nodeName) { + dssNode.removeDependency(nodeName); + } + + @Override + public List getDependencys() { + return dssNode.getDependencys(); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowWithContextImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowWithContextImpl.java new file mode 100644 index 000000000..830125d44 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/entity/WorkflowWithContextImpl.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.entity; + + +public class WorkflowWithContextImpl extends WorkflowImpl { + + private String contextID; + + public String getContextID() { + return contextID; + } + + public void setContextID(String contextID) { + this.contextID = contextID; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/flow2json/FlowToJsonParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/flow2json/FlowToJsonParser.java new file mode 100644 index 000000000..3b9185c14 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/flow2json/FlowToJsonParser.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.flow2json; + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSJsonFlow; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; + +/** + * Used to parse a workflow to DSSJsonFlow, it is useful when we want to use + * Workflow java SDK to create workflows and store it to db. + */ +public interface FlowToJsonParser { + + /** + * TODO wait for implementation. + * @param workflow + * @return + */ + DSSJsonFlow parse(Workflow workflow); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/AbstractJsonToFlowParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/AbstractJsonToFlowParser.java new file mode 100644 index 000000000..1ff6e4250 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/AbstractJsonToFlowParser.java @@ -0,0 +1,88 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow; + +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowImpl; +import com.webank.wedatasphere.dss.workflow.core.json2flow.parser.WorkflowParser; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.stream.Collectors; +import org.springframework.beans.BeanUtils; + + +public abstract class AbstractJsonToFlowParser implements JsonToFlowParser { + + private List workflowParsers; + + public List getWorkflowParsers() { + return workflowParsers; + } + + public synchronized void addWorkflowParsers(List workflowParsers) { + if(workflowParsers == null || workflowParsers.isEmpty()) { + return; + } + workflowParsers.forEach(WorkflowParser::init); + if(this.workflowParsers == null) { + this.workflowParsers = new ArrayList<>(); + } + this.workflowParsers.addAll(workflowParsers); + this.workflowParsers = this.workflowParsers.stream().sorted(Comparator.comparingInt(WorkflowParser::getOrder)).collect(Collectors.toList()); + } + + @Override + public void init() { + if(workflowParsers == null) { + workflowParsers = ClassUtils.getInstances(WorkflowParser.class); + } else { + workflowParsers.addAll(ClassUtils.getInstances(WorkflowParser.class)); + } + workflowParsers = workflowParsers.stream().sorted(Comparator.comparingInt(WorkflowParser::getOrder)).collect(Collectors.toList()); + workflowParsers.forEach(WorkflowParser::init); + } + + @Override + public Workflow parse(DSSFlow dssFlow) { + Workflow workflow = createWorkflow(); + BeanUtils.copyProperties(dssFlow, workflow, "children"); + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(dssFlow.getFlowJson()).getAsJsonObject(); + for(WorkflowParser workflowParser : workflowParsers) { + workflow = workflowParser.parse(jsonObject, workflow); + } + if(dssFlow.getChildren() != null) { + Workflow parentWorkflow = workflow; + List children = dssFlow.getChildren().stream().map(childFlow -> { + Workflow childWorkflow = parse(childFlow); + if(parentWorkflow instanceof WorkflowImpl) { + ((WorkflowImpl) childWorkflow).setParentWorkflow((WorkflowImpl) parentWorkflow); + } + return childWorkflow; + }).collect(Collectors.toList()); + workflow.setChildren(children); + } + return workflow; + } + + protected abstract Workflow createWorkflow(); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/JsonToFlowParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/JsonToFlowParser.java new file mode 100644 index 000000000..c21078ec8 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/JsonToFlowParser.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow; + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; + + +public interface JsonToFlowParser { + + void init(); + + Workflow parse(DSSFlow dssFlow); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowContextParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowContextParser.java new file mode 100644 index 000000000..2e9a15665 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowContextParser.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.google.gson.JsonObject; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowWithContextImpl; +import org.springframework.beans.BeanUtils; + + +public class WorkflowContextParser implements WorkflowParser { + + @Override + public Workflow parse(JsonObject flowJson, Workflow workflow) { + if (flowJson.has("contextID")) { + String contextID = flowJson.get("contextID").getAsString(); + WorkflowWithContextImpl workflowWithContext = new WorkflowWithContextImpl(); + BeanUtils.copyProperties(workflow, workflowWithContext); + workflowWithContext.setContextID(contextID); + return workflowWithContext; + } else { + return workflow; + } + } + + @Override + public int getOrder() { + return 10; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowDAGParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowDAGParser.java new file mode 100644 index 000000000..21a4c8ef8 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowDAGParser.java @@ -0,0 +1,102 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.google.gson.Gson; +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.reflect.TypeToken; +import com.webank.wedatasphere.dss.common.entity.node.DSSEdge; +import com.webank.wedatasphere.dss.common.entity.node.DSSEdgeDefault; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.entity.node.DSSNodeDefault; +import com.webank.wedatasphere.dss.common.utils.ClassUtils; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowImpl; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNodeEdge; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNodeEdgeImpl; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; + + +public class WorkflowDAGParser implements WorkflowParser { + + private List workflowNodeParsers; + + public synchronized void addWorkflowNodeParser(WorkflowNodeParser workflowNodeParser) { + if(workflowNodeParsers == null) { + workflowNodeParsers = new ArrayList<>(); + } + this.workflowNodeParsers.add(workflowNodeParser); + } + + protected List getWorkflowNodeParsers() { + return workflowNodeParsers; + } + + @Override + public void init() { + if(workflowNodeParsers == null) { + workflowNodeParsers = ClassUtils.getInstances(WorkflowNodeParser.class); + } else { + workflowNodeParsers.addAll(ClassUtils.getInstances(WorkflowNodeParser.class)); + } + } + + @Override + public Workflow parse(JsonObject flowJson, Workflow workflow) { + JsonArray nodeJsonArray = flowJson.getAsJsonArray("nodes"); + Gson gson = DSSCommonUtils.COMMON_GSON; + List dwsNodes = gson.fromJson(nodeJsonArray, new TypeToken>() { + }.getType()); + List workflowNodeList = new ArrayList<>(); + List workflowNodeEdgeList = new ArrayList<>(); + if (null != dwsNodes) { + for (DSSNode dwsNode : dwsNodes) { + Optional firstNodeParser = workflowNodeParsers.stream() + .filter(p -> p.ifNodeCanParse(dwsNode)) + .min((p1, p2) -> p2.getOrder() - p1.getOrder()); + WorkflowNode workflowNode = firstNodeParser.orElseThrow(() -> new IllegalArgumentException("NodeParser个数应该大于0")).parseNode(dwsNode); + workflowNodeList.add(workflowNode); + } + } + JsonArray edgeJsonArray = flowJson.getAsJsonArray("edges"); + List dwsEdges = gson.fromJson(edgeJsonArray, new TypeToken>() { + }.getType()); + if (dwsEdges != null){ + for (DSSEdge dwsEdge : dwsEdges) { + WorkflowNodeEdge workflowNodeEdge = new WorkflowNodeEdgeImpl(); + workflowNodeEdge.setDSSEdge(dwsEdge); + workflowNodeEdgeList.add(workflowNodeEdge); + } + } + if(workflow instanceof WorkflowImpl) { + WorkflowImpl workflow1 = (WorkflowImpl) workflow; + workflow1.setWorkflowNodeEdges(workflowNodeEdgeList); + workflow1.setWorkflowNodes(workflowNodeList); + } + return workflow; + } + + @Override + public int getOrder() { + return 0; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowDependencyParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowDependencyParser.java new file mode 100644 index 000000000..0ce6158f9 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowDependencyParser.java @@ -0,0 +1,93 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.google.gson.JsonObject; +import com.webank.wedatasphere.dss.workflow.core.constant.WorkflowConstant; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowImpl; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNodeEdge; +import java.util.ArrayList; +import java.util.List; +import org.apache.commons.lang.StringUtils; + + +public class WorkflowDependencyParser implements WorkflowParser { + + @Override + public Workflow parse(JsonObject flowJson, Workflow workflow) { + if(!(workflow instanceof WorkflowImpl)) { + return workflow; + } + //1.设置依赖，对于多个flowTuning，已经设置的就跳过 + WorkflowImpl realWorkflow = (WorkflowImpl) workflow; + workflow.getWorkflowNodes() + .forEach(node -> setDependencies(node, workflow.getWorkflowNodes(), workflow.getWorkflowNodeEdges())); + //2.设置usreproxy + setProxyUser(realWorkflow); + return workflow; + } + + private void setDependencies(WorkflowNode node, List workflowNodes, List flowEdges){ + //设置过的flow不在进行设置和解析 + if(node.getDependencys()!= null && !node.getDependencys().isEmpty()) { + return; + } + List dependencies = resolveDependencies(node, workflowNodes ,flowEdges); + dependencies.forEach(node::addDependency); + } + + private List resolveDependencies(WorkflowNode node, List workflowNodes, List flowEdges) { + List dependencies = new ArrayList<>(); + flowEdges.forEach(edge -> { + if (edge.getDSSEdge().getTarget().equals(node.getId())) { + dependencies.add(workflowNodes.stream().filter(n ->edge.getDSSEdge().getSource().equals(n.getId())).findFirst().get().getName()); + } + }); + + return dependencies; + } + + private String getProxyUser(Workflow workflow) { + if(workflow.getFlowProperties() == null) { + return null; + } + StringBuilder sb = new StringBuilder(); + workflow.getFlowProperties().forEach( map -> { + Object value = map.get(WorkflowConstant.PROXY_USER); + if(value != null && StringUtils.isNotBlank(value.toString())) { + sb.append(value.toString()); + } + }); + return sb.toString(); + } + + private void setProxyUser(WorkflowImpl workflow) { + String proxyUser = getProxyUser(workflow); + if(StringUtils.isNotBlank(proxyUser)) { + workflow.getWorkflowNodes().forEach(node -> node.getDSSNode().setUserProxy(proxyUser)); + workflow.setUserProxy(proxyUser); + } + } + + @Override + public int getOrder() { + return 20; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowNodeParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowNodeParser.java new file mode 100644 index 000000000..c74816e1f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowNodeParser.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import com.webank.wedatasphere.dss.workflow.core.order.Order; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; + +public interface WorkflowNodeParser extends Order { + + WorkflowNode parseNode(DSSNode dwsNode); + + Boolean ifNodeCanParse(DSSNode dwsNode); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowNodeParserImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowNodeParserImpl.java new file mode 100644 index 000000000..b8b46e086 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowNodeParserImpl.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNode; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowNodeImpl; + + +public class WorkflowNodeParserImpl implements WorkflowNodeParser { + @Override + public WorkflowNode parseNode(DSSNode dwsNode) { + WorkflowNodeImpl node = new WorkflowNodeImpl(); + node.setDSSNode(dwsNode); + return node; + } + + @Override + public Boolean ifNodeCanParse(DSSNode dwsNode) { + return true; + } + + @Override + public int getOrder() { + return 0; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowParser.java new file mode 100644 index 000000000..0d8f4ab25 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowParser.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.google.gson.JsonObject; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.order.Order; + +public interface WorkflowParser extends Order { + + default void init() {} + + Workflow parse(JsonObject flowJson, Workflow workflow); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowPropsParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowPropsParser.java new file mode 100644 index 000000000..5c47eeec9 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/json2flow/parser/WorkflowPropsParser.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.json2flow.parser; + +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.reflect.TypeToken; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.workflow.core.entity.Workflow; +import com.webank.wedatasphere.dss.workflow.core.entity.WorkflowImpl; +import java.util.List; +import java.util.Map; + + +public class WorkflowPropsParser implements WorkflowParser { + + @Override + public Workflow parse(JsonObject flowJson, Workflow workflow) { + JsonArray proJsonArray = flowJson.getAsJsonArray("props"); + List> props = DSSCommonUtils.COMMON_GSON.fromJson(proJsonArray, new TypeToken>>() { + }.getType()); + JsonArray resourcesJsonArray = flowJson.getAsJsonArray("resources"); + List resources = DSSCommonUtils.COMMON_GSON.fromJson(resourcesJsonArray, new TypeToken>() { + }.getType()); + if(workflow instanceof WorkflowImpl) { + WorkflowImpl workflow1 = (WorkflowImpl) workflow; + workflow1.setFlowResources(resources); + workflow1.setFlowProperties(props); + } + return workflow; + } + + @Override + public int getOrder() { + return 8; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/order/Order.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/order/Order.java new file mode 100644 index 000000000..791e66cc0 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-sdk/src/main/java/com/webank/wedatasphere/dss/workflow/core/order/Order.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.core.order; + +/** + * order接口用于标识parser和tunning，hook的调用顺序 + */ +public interface Order { + + int getOrder(); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/pom.xml new file mode 100644 index 000000000..28bf62b8b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/pom.xml @@ -0,0 +1,214 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-workflow-server + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + com.webank.wedatasphere.linkis + linkis-mybatis + ${linkis.version} + provided + + + linkis-bml-client + + + gson + com.google.code.gson + + + linkis-common + com.webank.wedatasphere.linkis + + + commons-beanutils + commons-beanutils + + + commons-collections + commons-collections + + + commons-logging + commons-logging + + + json4s-jackson_2.11 + org.json4s + + + com.webank.wedatasphere.linkis + ${linkis.version} + + + + com.webank.wedatasphere.linkis + linkis-rpc + ${linkis.version} + provided + + + spring-cloud-starter-netflix-eureka-client + org.springframework.cloud + + + + + + com.webank.wedatasphere.dss + dss-contextservice + ${dss.version} + + + linkis-common + com.webank.wedatasphere.linkis + + + commons-text + org.apache.commons + + + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + + + com.webank.wedatasphere.dss + dss-standard-common + ${dss.version} + + + com.webank.wedatasphere.dss + dss-scheduler-appconn + ${dss.version} + compile + + + commons-collections + commons-collections + + + jackson-databind + com.fasterxml.jackson.core + + + + + + com.webank.wedatasphere.dss + dss-appconn-manager-client + ${dss.version} + + + com.webank.wedatasphere.dss + dss-orchestrator-common + ${dss.version} + compile + + + com.webank.wedatasphere.dss + dss-sender-service + ${dss.version} + provided + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-assembly-plugin + 2.3 + false + + + make-assembly + package + + single + + + + src/main/assembly/distribution.xml + + + + + + false + out + false + false + + src/main/assembly/distribution.xml + + + + + + + src/main/java + + **/*.xml + + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/assembly/distribution.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/assembly/distribution.xml new file mode 100644 index 000000000..abcfce7c6 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/assembly/distribution.xml @@ -0,0 +1,43 @@ + + + + dss-workflow-server + + dir + + true + dss-workflow-server + + + + + + lib + true + true + false + true + true + + + + + + diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/DefaultWorkFlowManager.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/DefaultWorkFlowManager.java new file mode 100644 index 000000000..78e7a0ad7 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/DefaultWorkFlowManager.java @@ -0,0 +1,235 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow; + + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.appconn.scheduler.SchedulerAppConn; +import com.webank.wedatasphere.dss.common.entity.project.DSSProject; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.common.utils.ZipHelper; +import com.webank.wedatasphere.dss.orchestrator.common.entity.DSSOrchestration; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestConvertOrchestrations; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.operation.DSSToRelConversionOperation; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.DSSToRelConversionRequestRef; +import com.webank.wedatasphere.dss.orchestrator.converter.standard.ref.ProjectToRelConversionRequestRefImpl; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AppConnRefFactoryUtils; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import com.webank.wedatasphere.dss.workflow.entity.DSSFlowImportParam; +import com.webank.wedatasphere.dss.workflow.io.export.WorkFlowExportService; +import com.webank.wedatasphere.dss.workflow.io.input.MetaInputService; +import com.webank.wedatasphere.dss.workflow.io.input.WorkFlowInputService; +import com.webank.wedatasphere.dss.workflow.service.BMLService; +import com.webank.wedatasphere.dss.workflow.service.DSSFlowService; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import java.io.IOException; +import java.io.InputStream; +import java.util.Collections; +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +@Component +public class DefaultWorkFlowManager implements WorkFlowManager { + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private DSSFlowService flowService; + + @Autowired + private WorkFlowInputService workFlowInputService; + + @Autowired + private WorkFlowExportService workFlowExportService; + + @Autowired + private BMLService bmlService; + + @Autowired + private MetaInputService metaInputService; + + private Logger LOGGER = LoggerFactory.getLogger(this.getClass()); + + @Override + public DSSFlow createWorkflow(String userName, + String flowName, + String contextIdStr, + String description, + Long parentFlowId, + String uses, + List linkedAppConnNames, + List dssLabels) throws DSSErrorException, JsonProcessingException { + DSSFlow dssFlow = new DSSFlow(); + dssFlow.setName(flowName); + dssFlow.setDescription(description); + dssFlow.setCreator(userName); + dssFlow.setCreateTime(new Date()); + dssFlow.setState(false); + dssFlow.setSource("create by user"); + dssFlow.setUses(uses); + + String appConnJson = BDPJettyServerHelper.jacksonJson().writeValueAsString(linkedAppConnNames); + dssFlow.setLinkedAppConnNames(appConnJson); + Map dssLabelList = new HashMap<>(); + if (null != dssLabels) { + dssLabels.stream().map(a->a.getValue()).forEach(a->{ + dssLabelList.put(EnvDSSLabel.DSS_ENV_LABEL_KEY,a.get(EnvDSSLabel.DSS_ENV_LABEL_KEY)); + }); + } else { + dssLabelList.put(EnvDSSLabel.DSS_ENV_LABEL_KEY, DSSCommonUtils.ENV_LABEL_VALUE_DEV); + } + dssFlow.setDssLabels(BDPJettyServerHelper.jacksonJson().writeValueAsString(dssLabelList)); + //-1是rpc传递消息不支持null,所有父工作流都是-1。 + if (parentFlowId == null || parentFlowId == -1) { + dssFlow.setRootFlow(true); + dssFlow.setRank(0); + dssFlow.setHasSaved(true); + dssFlow = flowService.addFlow(dssFlow, contextIdStr); + } else { + dssFlow.setRootFlow(false); + Integer rank = flowService.getParentRank(parentFlowId); + // TODO: 2019/6/3 并发问题考虑for update + dssFlow.setRank(rank + 1); + dssFlow.setHasSaved(true); + dssFlow = flowService.addSubFlow(dssFlow, parentFlowId, contextIdStr); + } + return dssFlow; + } + + @Override + public DSSFlow copyRootflowWithSubflows(String userName, long rootFlowId, String workspaceName, String projectName, String contextIdStr, String version, String description) throws DSSErrorException, IOException { + + return flowService.copyRootFlow(rootFlowId, userName, workspaceName, projectName, version,contextIdStr); + + } + + @Override + public DSSFlow queryWorkflow(String userName, Long rootFlowId) throws DSSErrorException { + DSSFlow dssFlow = flowService.getFlowWithJsonAndSubFlowsByID(rootFlowId); + if (!dssFlow.getCreator().equals(userName)) { + throw new DSSErrorException(100089, "Workflow can not be query by others"); + } + return dssFlow; + } + + @Override + public void updateWorkflow(String userName, Long flowId, String flowName, String description, String uses) throws DSSErrorException { + DSSFlow dssFlow = new DSSFlow(); + dssFlow.setId(flowId); + dssFlow.setName(flowName); + dssFlow.setDescription(description); + dssFlow.setUses(uses); + flowService.updateFlowBaseInfo(dssFlow); + } + + @Override + public void deleteWorkflow(String userName, Long flowId) throws DSSErrorException { + DSSFlow dssFlow = flowService.getFlowByID(flowId); + if (dssFlow.getCreator().equals(userName)) { + flowService.batchDeleteFlow(Collections.singletonList(flowId)); + } else { + throw new DSSErrorException(100088, "Workflow can not be deleted by others"); + } + } + + @Override + public Map exportWorkflow(String userName, Long flowId, Long dssProjectId, String projectName, Workspace workspace,List dssLabels) throws Exception { + DSSFlow dssFlow = flowService.getFlowByID(flowId); + String exportPath = workFlowExportService.exportFlowInfo(dssProjectId, projectName, flowId, userName, workspace,dssLabels); + InputStream inputStream = bmlService.readLocalResourceFile(userName, exportPath); + Map resultMap = bmlService.upload(userName, inputStream, dssFlow.getName() + ".export", projectName); + return resultMap; + } + + @Override + public List importWorkflow(String userName, + String resourceId, + String bmlVersion, + DSSFlowImportParam dssFlowImportParam) throws DSSErrorException, IOException { + + //todo download workflow bml file contains flowInfo and flowRelationInfo + String inputZipPath = IoUtils.generateIOPath(userName, dssFlowImportParam.getProjectName(), dssFlowImportParam.getProjectName() + ".zip"); + bmlService.downloadToLocalPath(userName, resourceId, bmlVersion, inputZipPath); + String inputPath = ZipHelper.unzip(inputZipPath); + //导入工作流数据库信息 + List dssFlows = metaInputService.inputFlow(inputPath); + //导入工作流关系信息 + List dwsFlowRelations = metaInputService.inputFlowRelation(inputPath); + + List dwsFlowRelationList = workFlowInputService.persistenceFlow(dssFlowImportParam.getProjectID(), + dssFlowImportParam.getUserName(), + dssFlows, + dwsFlowRelations, + dssFlowImportParam.getSourceEnv()); + List rootFlows = dwsFlowRelationList.stream().filter(DSSFlow::getRootFlow).collect(Collectors.toList()); + for (DSSFlow rootFlow : rootFlows) { + workFlowInputService.inputWorkFlow(dssFlowImportParam.getUserName(), + dssFlowImportParam.getWorkspaceName(), + rootFlow, + dssFlowImportParam.getVersion(), + dssFlowImportParam.getProjectName(), + inputPath, null, dssFlowImportParam.getWorkspace(), dssFlowImportParam.getOrcVersion(), + dssFlowImportParam.getContextId()); + } + return rootFlows; + } + + @Override + public ResponseOperateOrchestrator convertWorkflow(RequestConvertOrchestrations requestConversionWorkflow) throws DSSErrorException { + //TODO try to optimize it by select db in batch. + List flows = requestConversionWorkflow.getOrcAppIds().stream().map(flowService::getFlowWithJsonAndSubFlowsByID).collect(Collectors.toList()); + SchedulerAppConn appConn = AppConnManager.getAppConnManager().getAppConn(SchedulerAppConn.class); +// List appInstances = appConn.getAppDesc().getAppInstancesByLabels(requestConversionWorkflow.getDSSLabels()); + AppInstance schedulerInstance = appConn.getAppDesc().getAppInstances().get(0); + DSSToRelConversionOperation operation = appConn.getOrCreateWorkflowConversionStandard() + .getDSSToRelConversionService(schedulerInstance).getDSSToRelConversionOperation(); + DSSToRelConversionRequestRef requestRef = AppConnRefFactoryUtils.newAppConnRef(DSSToRelConversionRequestRef.class, appConn.getAppDesc().getAppName()); + if(requestRef instanceof ProjectToRelConversionRequestRefImpl) { + ProjectToRelConversionRequestRefImpl relConversionRequestRef = (ProjectToRelConversionRequestRefImpl) requestRef; + relConversionRequestRef.setDSSProject((DSSProject) requestConversionWorkflow.getProject()); + relConversionRequestRef.setDSSOrcList(flows); + relConversionRequestRef.setUserName(requestConversionWorkflow.getUserName()); + relConversionRequestRef.setWorkspace((Workspace) requestConversionWorkflow.getWorkspace()); + } + try{ + ResponseRef responseRef = operation.convert(requestRef); + if(responseRef.isFailed()) { + return ResponseOperateOrchestrator.failed(responseRef.getErrorMsg()); + } + return ResponseOperateOrchestrator.success(); + }catch (Exception e){ + logger.error("convertWorkflow error:",e); + return ResponseOperateOrchestrator.failed(e.getMessage()); + } + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/WorkFlowManager.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/WorkFlowManager.java new file mode 100644 index 000000000..b3f682b4a --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/WorkFlowManager.java @@ -0,0 +1,100 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow; + + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestConvertOrchestrations; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseOperateOrchestrator; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.entity.DSSFlowImportParam; +import java.io.IOException; +import java.util.List; +import java.util.Map; + +public interface WorkFlowManager { + + /** + * create a new workflow + * @param userName + * @param workflowName + * @param contextIDStr + * @param description + * @param parentFlowID + * @param uses + * @return + */ + DSSFlow createWorkflow(String userName, + String workflowName, + String contextIDStr, + String description, + Long parentFlowID, + String uses, + List linkedAppConnNames, + List dssLabels + ) throws DSSErrorException, JsonProcessingException; + + + /** + * 根据已有的flow内容，拷贝一份新的flow,区分BML文件，用于绑定不同的版本 + * @param userName + * @param rootFlowId + * @param contextIDStr + * @param description + * @return + * @throws DSSErrorException + */ + DSSFlow copyRootflowWithSubflows(String userName, + long rootFlowId, + String workspaceName, + String projectName, + String contextIDStr, + String version, + String description) throws DSSErrorException, IOException; + + DSSFlow queryWorkflow(String userName, Long rootFlowId) throws DSSErrorException; + + + void updateWorkflow(String userName, + Long flowID, + String flowName, + String description, + String uses) throws DSSErrorException; + + + void deleteWorkflow(String userName, + Long flowID) throws DSSErrorException; + + + Map exportWorkflow(String userName, + Long flowID, + Long dssProjectId, + String projectName, + Workspace workspace, + List dssLabels) throws Exception; + + List importWorkflow(String userName, + String resourceId, + String bmlVersion, + DSSFlowImportParam dssFlowImportParam) throws Exception; + + public ResponseOperateOrchestrator convertWorkflow(RequestConvertOrchestrations requestConversionWorkflow) throws DSSErrorException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/conf/WorkflowServerSpringConfiguration.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/conf/WorkflowServerSpringConfiguration.java new file mode 100644 index 000000000..35cca8dbf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/conf/WorkflowServerSpringConfiguration.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.conf; + +import com.webank.wedatasphere.dss.workflow.service.PublishService; +import com.webank.wedatasphere.dss.workflow.service.impl.PublishServiceImpl; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + + +@Configuration +public class WorkflowServerSpringConfiguration { + + @Bean + @ConditionalOnMissingBean + public PublishService createPublishService() { + return new PublishServiceImpl(); + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/constant/DSSWorkFlowConstant.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/constant/DSSWorkFlowConstant.java new file mode 100644 index 000000000..86d859c82 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/constant/DSSWorkFlowConstant.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.constant; + +import com.google.common.collect.Interner; +import com.google.common.collect.Interners; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; +import com.webank.wedatasphere.linkis.common.conf.CommonVars$; + + +public class DSSWorkFlowConstant { + public static final CommonVars DSS_EXPORT_ENV = CommonVars.apply("wds.dss.server.export.env", "DEV"); + public static final CommonVars DSS_IMPORT_ENV = CommonVars.apply("wds.dss.server.import.env", "PROD"); + public static final String PUBLISH_FLOW_REPORT_FORMATE = "工作流名:%s,版本号:%s，工作流内容为空,请自行修改或者删除"; + public static final Interner saveFlowLock = Interners.newWeakInterner(); + public static final CommonVars CACHE_TIMEOUT = CommonVars$.MODULE$.apply("wds.dss.server.cache.timeout",1000 * 60 * 60); + public static final CommonVars PUBLISH_TIMEOUT = CommonVars$.MODULE$.apply("wds.dss.server.publish.timeout",60 * 10); + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/constant/WorkflowSpringConfiguration.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/constant/WorkflowSpringConfiguration.java new file mode 100644 index 000000000..9b310b32d --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/constant/WorkflowSpringConfiguration.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.constant; + +import com.webank.wedatasphere.dss.workflow.service.PublishService; +import com.webank.wedatasphere.dss.workflow.service.impl.PublishServiceImpl; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + + +@Configuration +public class WorkflowSpringConfiguration { + + @Bean + @ConditionalOnMissingBean + public PublishService createPublishService() { + return new PublishServiceImpl(); + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/DSSCSHelper.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/DSSCSHelper.java new file mode 100644 index 000000000..251688809 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/DSSCSHelper.java @@ -0,0 +1,70 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.cs; + +import com.google.gson.Gson; +import com.google.gson.JsonObject; +import com.webank.wedatasphere.linkis.cs.client.service.CSTableService; +import com.webank.wedatasphere.linkis.cs.common.entity.metadata.CSTable; +import com.webank.wedatasphere.linkis.cs.common.entity.source.ContextKeyValue; +import com.webank.wedatasphere.linkis.cs.common.exception.CSErrorException; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.List; + + +public class DSSCSHelper { + + private final static Logger logger = LoggerFactory.getLogger(DSSCSHelper.class); + + private final static Gson gson = new Gson(); + + public static List getTableContextKeyValueList(String contextIDStr, String nodeName) { + try { + return CSTableService.getInstance().searchUpstreamTableKeyValue(contextIDStr, nodeName); + } catch (CSErrorException e) { + logger.error("Failed to get getTableContextKeyValueList: " + nodeName, e); + } + + logger.info("upstream tmp table is null"); + return null; + } + + + public static CSTable getCSTable(String contextIDStr, String contextKeyStr) { + + + try { + return CSTableService.getInstance().getCSTable(contextIDStr, contextKeyStr); + } catch (CSErrorException e) { + logger.error("Failed to get CSTable contextkey: " + contextKeyStr, e); + } + logger.info("This contextKey{} has no csTable ", contextKeyStr); + return null; + } + + public static String getContextIDStrByJson(String flowJson) { + + JsonObject jsonObject = gson.fromJson(flowJson, JsonObject.class); + if (!jsonObject.has(CSCommonUtils.CONTEXT_ID_STR) || null == jsonObject.get(CSCommonUtils.CONTEXT_ID_STR)){ + return null; + } + return jsonObject.get(CSCommonUtils.CONTEXT_ID_STR).getAsString(); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/service/CSTableService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/service/CSTableService.java new file mode 100644 index 000000000..60a0b5a91 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/service/CSTableService.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.cs.service; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + +import java.util.List; +import java.util.Map; + + +public interface CSTableService { + + List> queryTables(String dbName, String contextIDStr, String nodeName) throws DSSErrorException; + + List> queryTableMeta(String dbName, String contextIDStr, String contextKeyStr) throws DSSErrorException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/service/CSTableServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/service/CSTableServiceImpl.java new file mode 100644 index 000000000..f5d2f56a3 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/cs/service/CSTableServiceImpl.java @@ -0,0 +1,88 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.cs.service; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.workflow.cs.DSSCSHelper; +import com.webank.wedatasphere.linkis.cs.client.utils.SerializeHelper; +import com.webank.wedatasphere.linkis.cs.common.entity.metadata.CSColumn; +import com.webank.wedatasphere.linkis.cs.common.entity.metadata.CSTable; +import com.webank.wedatasphere.linkis.cs.common.entity.source.ContextKeyValue; +import com.webank.wedatasphere.linkis.cs.common.entity.source.ContextValue; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + + +@Service +public class CSTableServiceImpl implements CSTableService { + + private final static Logger logger = LoggerFactory.getLogger(CSTableServiceImpl.class); + + + @Override + public List> queryTables(String dbName, String contextIDStr, String nodeName) throws DSSErrorException { + List contextKeyValueList = DSSCSHelper.getTableContextKeyValueList(contextIDStr, nodeName); + List> tables = new ArrayList<>(); + if (null == contextKeyValueList || contextKeyValueList.isEmpty()) { + return tables; + } + for(ContextKeyValue contextKeyValue : contextKeyValueList){ + Map tableNode = new HashMap<>(); + ContextValue contextValue = contextKeyValue.getContextValue(); + if (null != contextKeyValue && null != contextValue.getValue()) { + try { + CSTable table = (CSTable) (contextValue.getValue()) ; + tableNode.put("tableName", table.getName()); + tableNode.put("isView", table.isView()); + tableNode.put("databaseName", dbName); + tableNode.put("createdBy", table.getCreator()); + tableNode.put("createdAt", table.getCreateTime()); + //contextKey 需要序列化 + tableNode.put("contextKey", SerializeHelper.serializeContextKey(contextKeyValue.getContextKey())); + tables.add(tableNode); + } catch (Exception e) { + logger.error("Failed to get table ", e); + } + } + + } + return tables; + } + + @Override + public List> queryTableMeta(String dbName, String contextIDStr, String contextKeyStr) throws DSSErrorException { + CSTable csTable = DSSCSHelper.getCSTable(contextIDStr, contextKeyStr); + CSColumn[] columns = csTable.getColumns(); + List> responseTables = new ArrayList<>(); + for (CSColumn column : columns) { + Map columnNode = new HashMap<>(); + columnNode.put("columnName", column.getName()); + columnNode.put("columnType", column.getType()); + columnNode.put("columnComment", column.getComment()); + columnNode.put("partitioned", false); + responseTables.add(columnNode); + } + return responseTables; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/FlowMapper.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/FlowMapper.java new file mode 100644 index 000000000..3a8d9342b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/FlowMapper.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.dao; + + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import org.apache.ibatis.annotations.Param; +import org.apache.ibatis.annotations.Select; +import org.springframework.dao.DuplicateKeyException; + +import java.util.List; + + +public interface FlowMapper { + DSSFlow selectFlowByID(Long id); + + void insertFlow(DSSFlow dssFlow) throws DuplicateKeyException; + + void insertFlowRelation(@Param("flowID") Long flowID, @Param("parentFlowID") Long parentFlowID); + + void updateFlowBaseInfo(DSSFlow dssFlow) throws DuplicateKeyException; + + List selectSubFlowIDByParentFlowID(Long parentFlowID); + + List selectSavedSubFlowIDByParentFlowID(Long parentFlowID); + + void deleteFlowBaseInfo(Long flowID); + + void deleteFlowRelation(Long flowID); + + Long selectParentFlowIDByFlowID(Long flowID); + + Long getParentFlowID(Long flowID); + + List listFlowRelation(@Param("flowIDs") List flowIDs); + + void updateFlowInputInfo(DSSFlow dssFlow); + + DSSFlowRelation selectFlowRelation(@Param("flowID") Long flowID, @Param("parentFlowID") Long parentFlowID); + + @Select("select creator from dss_flow where id = #{flowId}") + String getCreatorById(@Param("flowId")Long flowId); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/InputRelationMapper.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/InputRelationMapper.java new file mode 100644 index 000000000..3d5bddfa3 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/InputRelationMapper.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.dao; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; + + +import com.webank.wedatasphere.dss.common.entity.InputRelation; +import org.apache.ibatis.annotations.Param; + +public interface InputRelationMapper { + + InputRelation selectInputRelation(@Param("type") String type, + @Param("sourceEnv") IOEnv sourceEnv, + @Param("sourceID") Long sourceID, + @Param("targetEnv") IOEnv targetEnv); + + void insertInputRelation(@Param("inputRelation") InputRelation inputRelation); + + void removeInputRelation(@Param("type") String type, + @Param("sourceEnv") IOEnv sourceEnv, + @Param("targetID") Long targetID, + @Param("targetEnv") IOEnv targetEnv); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/NodeInfoMapper.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/NodeInfoMapper.java new file mode 100644 index 000000000..3a950b7a4 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/NodeInfoMapper.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.dao; + + +import com.webank.wedatasphere.dss.workflow.entity.NodeGroup; +import com.webank.wedatasphere.dss.workflow.entity.NodeInfo; +import java.util.List; + + +public interface NodeInfoMapper { + + List listNodeGroups(); + + NodeInfo getWorkflowNodeByType(String nodeType); + +} + diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/flowMapper.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/flowMapper.xml new file mode 100644 index 000000000..507ed6b24 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/flowMapper.xml @@ -0,0 +1,141 @@ + + + + + + + + + + id,`name`,`state`,`source`,`description`,`create_time`,`creator`,`is_root_flow`,`rank`,`project_id`,`has_saved`,`uses`,`bml_version`,`resource_id`,`linked_appconn_names`,`dss_labels` + + + + + + + INSERT INTO dss_workflow () + VALUES + (#{id},#{name},#{state},#{source},#{description},#{createTime},#{creator},#{isRootFlow},#{rank},#{projectID},#{hasSaved},#{uses},#{bmlVersion},#{resourceId},#{linkedAppConnNames},#{dssLabels}) + + + + + + INSERT INTO dss_flow_relation (flow_id,parent_flow_id) + VALUES + (#{flowID},#{parentFlowID}) + + + + UPDATE dss_workflow + + name=#{name}, + description=#{description}, + has_saved=#{hasSaved}, + uses=#{uses}, + linked_appconn_names=#{linkedAppConnNames}, + + WHERE id =#{id} + + + + + + + + DELETE + FROM + dss_workflow + WHERE id = #{flowID} + + + + DELETE + FROM + dss_flow_relation + WHERE flow_id = #{flowID} + + + + + + + + + + + + + + + + UPDATE dss_workflow + + name=#{name}, + state = #{state}, + source = #{source}, + description=#{description}, + create_time=#{createTime}, + creator=#{creator}, + project_id=#{projectID}, + has_saved=#{hasSaved}, + uses=#{uses}, + resource_id=#{resourceId}, + bml_version=#{bmlVersion}, + linked_appconn_names=#{linkedAppConnNames}, + + WHERE id =#{id} + + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/inputRelationMapper.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/inputRelationMapper.xml new file mode 100644 index 000000000..ba2acc868 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/inputRelationMapper.xml @@ -0,0 +1,42 @@ + + + + + + + + + + + + insert into dss_input_relation values(#{inputRelation.id}, + #{inputRelation.type}, + #{inputRelation.sourceEnv}, + #{inputRelation.sourceID}, + #{inputRelation.targetEnv}, + #{inputRelation.targetID}) + + + + delete from dss_input_relation where type = #{type} and source_env = #{sourceEnv} and target_env = #{targetEnv} + and target_id = #{targetID} + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/nodeInfoMapper.xml b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/nodeInfoMapper.xml new file mode 100644 index 000000000..fb882c604 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/dao/impl/nodeInfoMapper.xml @@ -0,0 +1,142 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AbstractAppConnNode.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AbstractAppConnNode.java new file mode 100644 index 000000000..9c46da991 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AbstractAppConnNode.java @@ -0,0 +1,135 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + + + +import java.util.Map; + + +public abstract class AbstractAppConnNode { + + String projectName; + + long projectId; + + + String flowName; + + long flowId; + + String nodeName; + + String nodeId; + + String nodeType; + + Map jobContent; + + public AbstractAppConnNode(String projectName, long projectId, String flowName, long flowId, + String nodeName, String nodeTye, Map jobContent){ + this.projectId = projectId; + this.projectName = projectName; + this.flowName = flowName; + this.flowId = flowId; + this.nodeName = nodeName; + this.nodeType = nodeTye; + this.jobContent = jobContent; + } + + public AbstractAppConnNode(){ + + } + + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public void setProjectId(long projectId) { + this.projectId = projectId; + } + + public void setFlowName(String flowName) { + this.flowName = flowName; + } + + public void setFlowId(long flowId) { + this.flowId = flowId; + } + + + + public String getId() { + return this.nodeId; + } + + + public void setId(String id) { + this.nodeId = id; + } + + + public String getNodeType() { + return this.nodeType; + } + + + public void setNodeType(String nodeType) { + this.nodeType = nodeType; + } + + + public String getName() { + return this.nodeName; + } + + + public void setName(String name) { + this.nodeName = name; + } + + + public long getProjectId() { + return this.projectId; + } + + public String getProjectName() { + return this.projectName; + } + + + public String getFlowName() { + return this.flowName; + } + + + public long getFlowId() { + return this.flowId; + } + + + public Map getJobContent() { + return this.jobContent; + } + + + public void setJobContent(Map jobContent) { + this.jobContent = jobContent; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMap.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMap.java new file mode 100644 index 000000000..32aa40703 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMap.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +import java.util.List; + +public class AppMap { + public List getEdges() { + return edges; + } + + public void setEdges(List edges) { + this.edges = edges; + } + + public List getNodes() { + return nodes; + } + + public void setNodes(List nodes) { + this.nodes = nodes; + } + + List edges; + List nodes; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapEdge.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapEdge.java new file mode 100644 index 000000000..f44cd5fd2 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapEdge.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +public class AppMapEdge { + + private String source; + private String target; + private String sourceLocation; + private String targetLocation; + private String edgeType; + + public String getSource() { + return source; + } + + public void setSource(String source) { + this.source = source; + } + + public String getTarget() { + return target; + } + + public void setTarget(String target) { + this.target = target; + } + + public String getSourceLocation() { + return sourceLocation; + } + + public void setSourceLocation(String sourceLocation) { + this.sourceLocation = sourceLocation; + } + + public String getTargetLocation() { + return targetLocation; + } + + public void setTargetLocation(String targetLocation) { + this.targetLocation = targetLocation; + } + + public String getEdgeType() { + return edgeType; + } + + public void setEdgeType(String edgeType) { + this.edgeType = edgeType; + } +} + + diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapFlowNode.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapFlowNode.java new file mode 100644 index 000000000..f547014ee --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapFlowNode.java @@ -0,0 +1,104 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +import java.util.HashMap; + +public class AppMapFlowNode { + private String id; + private String desc; + private String businessTag; + private String appTag; + private String key; + private String title; + + private String isRootFlow; + + public String getIsRootFlow() { + return isRootFlow; + } + + public void setIsRootFlow(String isRootFlow) { + this.isRootFlow = isRootFlow; + } + + + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public String getDesc() { + return desc; + } + + public void setDesc(String desc) { + this.desc = desc; + } + + public String getBusinessTag() { + return businessTag; + } + + public void setBusinessTag(String businessTag) { + this.businessTag = businessTag; + } + + public String getAppTag() { + return appTag; + } + + public void setAppTag(String appTag) { + this.appTag = appTag; + } + + public String getKey() { + return key; + } + + public void setKey(String key) { + this.key = key; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public AppMapProjectInfo getProjectData() { + return projectData; + } + + public void setProjectData(AppMapProjectInfo projectData) { + this.projectData = projectData; + } + + private AppMapProjectInfo projectData; + private HashMap layout=new HashMap(){{ + put("height",120); + put("width",200); + put("x",""); + put("y","");} + }; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapProjectInfo.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapProjectInfo.java new file mode 100644 index 000000000..646a8ac89 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/AppMapProjectInfo.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +public class AppMapProjectInfo { + private long id; + private String desc; + private String title; + private boolean currentProject = false; + private String businessTag; + private String appTag; + + public long getId() { + return id; + } + + public void setId(long id) { + this.id = id; + } + + public String getDesc() { + return desc; + } + + public void setDesc(String desc) { + this.desc = desc; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public boolean isCurrentProject() { + return currentProject; + } + + public void setCurrentProject(boolean currentProject) { + this.currentProject = currentProject; + } + + public String getBusinessTag() { + return businessTag; + } + + public void setBusinessTag(String businessTag) { + this.businessTag = businessTag; + } + + public String getAppTag() { + return appTag; + } + + public void setAppTag(String appTag) { + this.appTag = appTag; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/CommonAppConnNode.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/CommonAppConnNode.java new file mode 100644 index 000000000..cf9f9a3da --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/CommonAppConnNode.java @@ -0,0 +1,45 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + + +public class CommonAppConnNode extends AbstractAppConnNode { + + private Map params; + + public CommonAppConnNode(String projectName, long projectId , String flowName, long flowId, String nodeName, + String type, Map jobContent){ + super(projectName, projectId, flowName, flowId, nodeName, type, jobContent); + } + + public CommonAppConnNode(){ + + } + + + public void setParams(Map params){ + this.params = params; + } + + public Map getParams(){ + return this.params; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/ContentLanguage.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/ContentLanguage.java new file mode 100644 index 000000000..481dcb735 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/ContentLanguage.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +public enum ContentLanguage { + en("en"), zh_CN("zh-CN"); + private String name; + + ContentLanguage(String name) { + this.name = name; + } + + public String getName() { + return this.name; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/DSSFlowImportParam.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/DSSFlowImportParam.java new file mode 100644 index 000000000..9d318552c --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/DSSFlowImportParam.java @@ -0,0 +1,108 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; + + +public class DSSFlowImportParam { + + String workspaceName; + Long projectID; + String projectName; + String version; + String orcVersion; + //TODO 目前只支持一个rootflow导入，只带了一个ContextID + String contextId; + IOEnv sourceEnv; + Workspace workspace; + + public String getUserName() { + return userName; + } + + public void setUserName(String userName) { + this.userName = userName; + } + + String userName; + + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public Long getProjectID() { + return projectID; + } + + public void setProjectID(Long projectID) { + this.projectID = projectID; + } + + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + public String getVersion() { + return version; + } + + public void setVersion(String version) { + this.version = version; + } + + public String getOrcVersion() { + return orcVersion; + } + + public void setOrcVersion(String orcVersion) { + this.orcVersion = orcVersion; + } + + public IOEnv getSourceEnv() { + return sourceEnv; + } + + public void setSourceEnv(IOEnv sourceEnv) { + this.sourceEnv = sourceEnv; + } + + public Workspace getWorkspace() { + return workspace; + } + + public void setWorkspace(Workspace workspace) { + this.workspace = workspace; + } + + public String getContextId() { + return contextId; + } + + public void setContextId(String contextId) { + this.contextId = contextId; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/InputRelation.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/InputRelation.java new file mode 100644 index 000000000..ff82ae885 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/InputRelation.java @@ -0,0 +1,78 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + + +import com.webank.wedatasphere.dss.common.entity.IOEnv; + +public class InputRelation { + + private Long id; + private String type; + private IOEnv sourceEnv; + private Long sourceID; + private IOEnv targetEnv; + private Long targetID; + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public IOEnv getSourceEnv() { + return sourceEnv; + } + + public void setSourceEnv(IOEnv sourceEnv) { + this.sourceEnv = sourceEnv; + } + + public Long getSourceID() { + return sourceID; + } + + public void setSourceID(Long sourceID) { + this.sourceID = sourceID; + } + + public IOEnv getTargetEnv() { + return targetEnv; + } + + public void setTargetEnv(IOEnv targetEnv) { + this.targetEnv = targetEnv; + } + + public Long getTargetID() { + return targetID; + } + + public void setTargetID(Long targetID) { + this.targetID = targetID; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeGroup.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeGroup.java new file mode 100644 index 000000000..f5bdd3adc --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeGroup.java @@ -0,0 +1,77 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +import java.io.Serializable; +import java.util.List; + +public class NodeGroup implements Serializable { + private Long id; + private String name; + private String nameEn; + private String description; + private Integer order; + private List nodes; + + public List getNodes() { + return nodes; + } + + public void setNodes(List nodes) { + this.nodes = nodes; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public String getNameEn() { + return nameEn; + } + + public void setNameEn(String nameEn) { + this.nameEn = nameEn; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Integer getOrder() { + return order; + } + + public void setOrder(Integer order) { + this.order = order; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeInfo.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeInfo.java new file mode 100644 index 000000000..2f86314b0 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeInfo.java @@ -0,0 +1,132 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +import java.io.Serializable; +import java.util.List; + + +public class NodeInfo implements Serializable { + private Integer id; + private String icon; + private String nodeType; + private Integer appConnId; + private String appConnName; + private Boolean submitToScheduler; //是否可以提交执行 + private Boolean enableCopy; + private Boolean shouldCreationBeforeNode; + private Boolean supportJump; + private String jumpUrl; + private String name; + private List nodeUis; + + public List getNodeUis() { + return nodeUis; + } + + public void setNodeUis(List nodeUis) { + this.nodeUis = nodeUis; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getIcon() { + return icon; + } + + public void setIcon(String icon) { + this.icon = icon; + } + + public String getNodeType() { + return nodeType; + } + + public void setNodeType(String nodeType) { + this.nodeType = nodeType; + } + + public Integer getAppConnId() { + return appConnId; + } + + public void setAppConnId(Integer appConnId) { + this.appConnId = appConnId; + } + + public String getAppConnName() { + return appConnName; + } + + public void setAppConnName(String appConnName) { + this.appConnName = appConnName; + } + + public Boolean getSubmitToScheduler() { + return submitToScheduler; + } + + public void setSubmitToScheduler(Boolean submitToScheduler) { + this.submitToScheduler = submitToScheduler; + } + + public Boolean getEnableCopy() { + return enableCopy; + } + + public void setEnableCopy(Boolean enableCopy) { + this.enableCopy = enableCopy; + } + + public Boolean getShouldCreationBeforeNode() { + return shouldCreationBeforeNode; + } + + public void setShouldCreationBeforeNode(Boolean shouldCreationBeforeNode) { + this.shouldCreationBeforeNode = shouldCreationBeforeNode; + } + + public Boolean getSupportJump() { + return supportJump; + } + + public void setSupportJump(Boolean supportJump) { + this.supportJump = supportJump; + } + + public String getJumpUrl() { + return jumpUrl; + } + + public void setJumpUrl(String jumpUrl) { + this.jumpUrl = jumpUrl; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUi.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUi.java new file mode 100644 index 000000000..d4f1f9234 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUi.java @@ -0,0 +1,187 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + + +import java.io.Serializable; +import java.util.List; + +public class NodeUi implements Serializable { + private Long id; + private String key; + private String description; + private String descriptionEn; + private String lableName; + private String lableNameEn; + private NodeUiType uiType; + private Boolean required; + private String value; + private String defaultValue; + private Boolean isHidden; + private String condition; + private Boolean isAdvanced; + private Integer order; + private Boolean isBaseInfo; + private Integer nodeMenuType;//1为右边栏 + private String position; + + private List nodeUiValidates; + + public List getNodeUiValidates() { + return nodeUiValidates; + } + + public void setNodeUiValidates(List nodeUiValidates) { + this.nodeUiValidates = nodeUiValidates; + } + + public String getPosition() { + return position; + } + + public void setPosition(String position) { + this.position = position; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getKey() { + return key; + } + + public void setKey(String key) { + this.key = key; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public String getDescriptionEn() { + return descriptionEn; + } + + public void setDescriptionEn(String descriptionEn) { + this.descriptionEn = descriptionEn; + } + + public String getLableName() { + return lableName; + } + + public void setLableName(String lableName) { + this.lableName = lableName; + } + + public String getLableNameEn() { + return lableNameEn; + } + + public void setLableNameEn(String lableNameEn) { + this.lableNameEn = lableNameEn; + } + + public NodeUiType getUiType() { + return uiType; + } + + public void setUiType(NodeUiType uiType) { + this.uiType = uiType; + } + + public Boolean getRequired() { + return required; + } + + public void setRequired(Boolean required) { + this.required = required; + } + + public String getValue() { + return value; + } + + public void setValue(String value) { + this.value = value; + } + + public String getDefaultValue() { + return defaultValue; + } + + public void setDefaultValue(String defaultValue) { + this.defaultValue = defaultValue; + } + + public Boolean getHidden() { + return isHidden; + } + + public void setHidden(Boolean hidden) { + isHidden = hidden; + } + + public String getCondition() { + return condition; + } + + public void setCondition(String condition) { + this.condition = condition; + } + + public Boolean getAdvanced() { + return isAdvanced; + } + + public void setAdvanced(Boolean advanced) { + isAdvanced = advanced; + } + + public Integer getOrder() { + return order; + } + + public void setOrder(Integer order) { + this.order = order; + } + + public Boolean getBaseInfo() { + return isBaseInfo; + } + + public void setBaseInfo(Boolean baseInfo) { + isBaseInfo = baseInfo; + } + + public Integer getNodeMenuType() { + return nodeMenuType; + } + + public void setNodeMenuType(Integer nodeMenuType) { + this.nodeMenuType = nodeMenuType; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiType.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiType.java new file mode 100644 index 000000000..9d66ac282 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiType.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +public enum NodeUiType { + Input, Text, Upload, Tag, Select, MultiSelect, Disable, Binding, MultiBinding +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidate.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidate.java new file mode 100644 index 000000000..83e73ba69 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidate.java @@ -0,0 +1,85 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +import java.io.Serializable; + +public class NodeUiValidate implements Serializable { + private Integer id; + private Integer uiId; + private NodeUiValidateType validateType; + private String validateRange; + private String errorMsg; + private String errorMsgEn; + private NodeUiValidateTrigger trigger; + + public NodeUiValidateTrigger getTrigger() { + return trigger; + } + + public void setTrigger(NodeUiValidateTrigger trigger) { + this.trigger = trigger; + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public Integer getUiId() { + return uiId; + } + + public void setUiId(Integer uiId) { + this.uiId = uiId; + } + + public NodeUiValidateType getValidateType() { + return validateType; + } + + public void setValidateType(NodeUiValidateType validateType) { + this.validateType = validateType; + } + + public String getValidateRange() { + return validateRange; + } + + public void setValidateRange(String validateRange) { + this.validateRange = validateRange; + } + + public String getErrorMsg() { + return errorMsg; + } + + public void setErrorMsg(String errorMsg) { + this.errorMsg = errorMsg; + } + + public String getErrorMsgEn() { + return errorMsgEn; + } + + public void setErrorMsgEn(String errorMsgEn) { + this.errorMsgEn = errorMsgEn; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidateTrigger.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidateTrigger.java new file mode 100644 index 000000000..27ddb4ee6 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidateTrigger.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +public enum NodeUiValidateTrigger { + /** + * blur 失去焦点事件 + *

+ * change 事件 + */ + blur, change +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidateType.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidateType.java new file mode 100644 index 000000000..f36a7a2f2 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/NodeUiValidateType.java @@ -0,0 +1,52 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity; + +public enum NodeUiValidateType { + /** + * 不校验，None不能和下面的校验类型搭配 + */ + None, + /** + * 整数类型 + */ + NumInterval, + /** + * 在这几个中 + */ + OFT, + /** + * 浮点数 + */ + FloatInterval, + /** + * 包含 + */ + Contains, + /** + * 正则 + */ + Regex, + /** + * 自定义函数..validate_range是函数名,由前台提供 + */ + Function, + /** + * 不为空 + */ + Required +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeGroupVO.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeGroupVO.java new file mode 100644 index 000000000..a27f099e7 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeGroupVO.java @@ -0,0 +1,77 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity.vo; + +import java.util.List; + +public class NodeGroupVO implements Comparable { + private Long id; + /** + * title是前台字段..后台自动根据请求头返回中英文字段 + * chilren是为了适配前台插件的字段 + */ + private String title; + private String description; + private Integer order; + + private List children; + + public List getChildren() { + return children; + } + + public void setChildren(List children) { + this.children = children; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public String getDescription() { + return description; + } + + public void setDescription(String description) { + this.description = description; + } + + public Integer getOrder() { + return order; + } + + public void setOrder(Integer order) { + this.order = order; + } + + @Override + public int compareTo(NodeGroupVO o) { + return this.order - o.order; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeInfoVO.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeInfoVO.java new file mode 100644 index 000000000..bc59c811f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeInfoVO.java @@ -0,0 +1,146 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity.vo; + +import java.util.List; + +public class NodeInfoVO { + + private Integer id; + //icon + private String image; + //nodeType + private String type; + private Integer applicationId; + private Boolean submitToScheduler; //是否可以提交执行 + private Boolean enableCopy; + private Boolean shouldCreationBeforeNode; + private Boolean supportJump; + private String jumpUrl; + //name + private String title; + //前台插件参数 默认false + private boolean editParam; + //前台插件参数 默认false + private boolean editBaseInfo; + + private List nodeUiVOS; + + public List getNodeUiVOS() { + return nodeUiVOS; + } + + public void setNodeUiVOS(List nodeUiVOS) { + this.nodeUiVOS = nodeUiVOS; + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getImage() { + return image; + } + + public void setImage(String image) { + this.image = image; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public Integer getApplicationId() { + return applicationId; + } + + public void setApplicationId(Integer applicationId) { + this.applicationId = applicationId; + } + + public Boolean getSubmitToScheduler() { + return submitToScheduler; + } + + public void setSubmitToScheduler(Boolean submitToScheduler) { + this.submitToScheduler = submitToScheduler; + } + + public Boolean getEnableCopy() { + return enableCopy; + } + + public void setEnableCopy(Boolean enableCopy) { + this.enableCopy = enableCopy; + } + + public Boolean getShouldCreationBeforeNode() { + return shouldCreationBeforeNode; + } + + public void setShouldCreationBeforeNode(Boolean shouldCreationBeforeNode) { + this.shouldCreationBeforeNode = shouldCreationBeforeNode; + } + + public Boolean getSupportJump() { + return supportJump; + } + + public void setSupportJump(Boolean supportJump) { + this.supportJump = supportJump; + } + + public String getJumpUrl() { + return jumpUrl; + } + + public void setJumpUrl(String jumpUrl) { + this.jumpUrl = jumpUrl; + } + + public String getTitle() { + return title; + } + + public void setTitle(String title) { + this.title = title; + } + + public boolean isEditParam() { + return editParam; + } + + public void setEditParam(boolean editParam) { + this.editParam = editParam; + } + + public boolean isEditBaseInfo() { + return editBaseInfo; + } + + public void setEditBaseInfo(boolean editBaseInfo) { + this.editBaseInfo = editBaseInfo; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeUiVO.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeUiVO.java new file mode 100644 index 000000000..cce659fdd --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeUiVO.java @@ -0,0 +1,178 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity.vo; + + + +import com.webank.wedatasphere.dss.workflow.entity.NodeUiType; + +import java.util.List; + +public class NodeUiVO implements Comparable { + private Long id; + private String key; + private String desc; + private String lableName; + private NodeUiType uiType; + private Boolean required; + private String value; + private String defaultValue; + private Boolean isHidden; + private String condition; + private Boolean isAdvanced; + private Integer order; + private Boolean isBaseInfo; + private Integer nodeMenuType;//1为右边栏 + private String position;//存放位置 nodes /nodes.params.startup/nodes.params.runtime + private List nodeUiValidateVOS; + + public List getNodeUiValidateVOS() { + return nodeUiValidateVOS; + } + + public void setNodeUiValidateVOS(List nodeUiValidateVOS) { + this.nodeUiValidateVOS = nodeUiValidateVOS; + } + + public String getPosition() { + return position; + } + + public void setPosition(String position) { + this.position = position; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getKey() { + return key; + } + + public void setKey(String key) { + this.key = key; + } + + public String getDesc() { + return desc; + } + + public void setDesc(String desc) { + this.desc = desc; + } + + public String getLableName() { + return lableName; + } + + public void setLableName(String lableName) { + this.lableName = lableName; + } + + public NodeUiType getUiType() { + return uiType; + } + + public void setUiType(NodeUiType uiType) { + this.uiType = uiType; + } + + public Boolean getRequired() { + return required; + } + + public void setRequired(Boolean required) { + this.required = required; + } + + public String getValue() { + return value; + } + + public void setValue(String value) { + this.value = value; + } + + public String getDefaultValue() { + return defaultValue; + } + + public void setDefaultValue(String defaultValue) { + this.defaultValue = defaultValue; + } + + public Boolean getHidden() { + return isHidden; + } + + public void setHidden(Boolean hidden) { + isHidden = hidden; + } + + public String getCondition() { + return condition; + } + + public void setCondition(String condition) { + this.condition = condition; + } + + public Boolean getAdvanced() { + return isAdvanced; + } + + public void setAdvanced(Boolean advanced) { + isAdvanced = advanced; + } + + public Integer getOrder() { + return order; + } + + public void setOrder(Integer order) { + this.order = order; + } + + public Boolean getBaseInfo() { + return isBaseInfo; + } + + public void setBaseInfo(Boolean baseInfo) { + isBaseInfo = baseInfo; + } + + public Integer getNodeMenuType() { + return nodeMenuType; + } + + public void setNodeMenuType(Integer nodeMenuType) { + this.nodeMenuType = nodeMenuType; + } + + @Override + public int compareTo(NodeUiVO o) { + if (!this.isBaseInfo.equals(o.isBaseInfo)) { + return Boolean.compare(o.isBaseInfo, this.isBaseInfo); + } + return this.order - o.order; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeUiValidateVO.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeUiValidateVO.java new file mode 100644 index 000000000..0e36ded73 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/entity/vo/NodeUiValidateVO.java @@ -0,0 +1,68 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.entity.vo; + + +import com.webank.wedatasphere.dss.workflow.entity.NodeUiValidateTrigger; +import com.webank.wedatasphere.dss.workflow.entity.NodeUiValidateType; + +public class NodeUiValidateVO implements Comparable { + + private NodeUiValidateType validateType; //前台的自定义校验函数名,对应validateType + private String validateRange; + private NodeUiValidateTrigger trigger; //blur change + private String message; + + public NodeUiValidateType getValidateType() { + return validateType; + } + + public void setValidateType(NodeUiValidateType validateType) { + this.validateType = validateType; + } + + public NodeUiValidateTrigger getTrigger() { + return trigger; + } + + public void setTrigger(NodeUiValidateTrigger trigger) { + this.trigger = trigger; + } + + public String getMessage() { + return message; + } + + public void setMessage(String message) { + this.message = message; + } + + public String getValidateRange() { + return validateRange; + } + + public void setValidateRange(String validateRange) { + this.validateRange = validateRange; + } + + + @Override + public int compareTo(NodeUiValidateVO o) { + //Required校验类型要在数组最前，否则前台无法渲染 + return o.validateType.ordinal() - this.validateType.ordinal(); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/DomainSupplier.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/DomainSupplier.java new file mode 100644 index 000000000..e668a2569 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/DomainSupplier.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.function; + +public interface DomainSupplier { + T get(P p); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/FunctionInvoker.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/FunctionInvoker.java new file mode 100644 index 000000000..4194f3994 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/FunctionInvoker.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.function; + +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.workflow.entity.AbstractAppConnNode; +import com.webank.wedatasphere.dss.workflow.service.WorkflowNodeService; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import java.util.Map; + + +@Component +public class FunctionInvoker { + + @Autowired + private WorkflowNodeService workflowNodeService; + + + private final Logger logger = LoggerFactory.getLogger(this.getClass()); + + public void nodeServiceFunction(String userName, Map requestBody, AbstractAppConnNode node, NodeServiceFunction function) throws IllegalAccessException, ExternalOperationFailedException, InstantiationException { + Map jobContent = null; + logger.info("appConn NodeService is exist"); + jobContent = function.accept(userName, workflowNodeService, node, requestBody); + node.setJobContent(jobContent); + + } + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/FunctionPool.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/FunctionPool.java new file mode 100644 index 000000000..b8a53ea20 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/FunctionPool.java @@ -0,0 +1,48 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package com.webank.wedatasphere.dss.workflow.function; + +import com.webank.wedatasphere.dss.workflow.entity.AbstractAppConnNode; +import com.webank.wedatasphere.dss.workflow.service.WorkflowNodeService; + +import java.util.Map; + + +public class FunctionPool { + + public static NodeServiceFunction deleteNode = (String userName, WorkflowNodeService nodeService, AbstractAppConnNode node, Map requestBody)->{ + node.setJobContent(requestBody); + nodeService.deleteNode(userName,node); + return null; + }; + + public static NodeServiceFunction createNode = (String userName, WorkflowNodeService nodeService, AbstractAppConnNode node, Map requestBody)-> + + { + node.setJobContent(requestBody); + return nodeService.createNode(userName, node); + }; + + + + public static NodeServiceFunction updateNode = (String userName, WorkflowNodeService nodeService, AbstractAppConnNode node, Map requestBody)-> + + { + node.setJobContent(requestBody); + return nodeService.updateNode(userName, node); + }; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/NodeServiceFunction.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/NodeServiceFunction.java new file mode 100644 index 000000000..ef4ee62bc --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/function/NodeServiceFunction.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.function; + +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.workflow.entity.AbstractAppConnNode; +import com.webank.wedatasphere.dss.workflow.service.WorkflowNodeService; + +import java.util.Map; + + +public interface NodeServiceFunction{ + Map accept(String userName, WorkflowNodeService nodeService, AbstractAppConnNode node, Map requestBody) throws ExternalOperationFailedException, IllegalAccessException, InstantiationException; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/MetaExportService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/MetaExportService.java new file mode 100644 index 000000000..87216241d --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/MetaExportService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export; + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; + +import java.io.IOException; +import java.util.List; + +public interface MetaExportService { + + + void exportFlowBaseInfo(List allDSSFlows, List allFlowRelations, String savePath) throws IOException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/MetaWriter.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/MetaWriter.java new file mode 100644 index 000000000..eb48b2b87 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/MetaWriter.java @@ -0,0 +1,173 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export; + +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.workflow.constant.DSSWorkFlowConstant; +import org.apache.commons.io.IOUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.stream.Collectors; + +public class MetaWriter { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + private final Class tClass; + private final String tableName; + private List datas = new ArrayList<>(); + private List ignoreFields = new ArrayList<>(); + private List fields = null; + private List comments = new ArrayList<>(); + private List table = new ArrayList<>(); + private String commentPrefix = "#"; + private String seperator = "|"; + + public MetaWriter(Class tClass, String tableName) { + this.tClass = tClass; + this.tableName = tableName; + } + + public static MetaWriter of(String tableName, Class tClass) { + return new MetaWriter<>(tClass, tableName); + } + + public MetaWriter ignore(String... fields) { + ignoreFields.addAll(Arrays.stream(fields).collect(Collectors.toList())); + return this; + } + + public MetaWriter data(List datas) { + this.datas.addAll(datas); + return this; + } + + public MetaWriter data(T data) { + this.datas.add(data); + return this; + } + + public MetaWriter comment(String... comments) { + this.comments.addAll(Arrays.stream(comments).collect(Collectors.toList())); + return this; + } + + /** + * 上传bml + * + * @return + */ + public InputStream write() { + writeComment(); + writeHead(); + writeBody(); + //table 添加换行符进行转流 + String tableStr = table.stream().reduce((a, b) -> a + "\n" + b).orElse("") + "\n"; + logger.info("\n" + tableStr); + return new ByteArrayInputStream(tableStr.getBytes()); + } + + /** + * 写入本地 + * + * @param outputStream + */ + public void write(OutputStream outputStream) throws IOException { + writeComment(); + writeHead(); + writeBody(); + IOUtils.writeLines(table, "\n", outputStream); + } + + private void writeBody() { + datas.forEach(DSSExceptionUtils.handling(this::writeBody)); + } + + private void writeBody(T t) throws NoSuchFieldException, IllegalAccessException { + ArrayList line = new ArrayList<>(); + for (String field : fields) { + Field declaredField = tClass.getDeclaredField(field); + declaredField.setAccessible(true); + Object o = declaredField.get(t); + if (o != null) { + line.add(o.toString()); + } else { + line.add(null); + } + } + table.add(reduce(line)); + } + + private String reduce(List strs) { + return strs.stream().reduce((a, b) -> a + seperator + b).orElse(""); + } + + /** + * 写表的头部 + * 1.过滤ignore的属性 + * 2.驼峰转mysql的 _ (暂时略过) + * 3.写入List table + */ + private void writeHead() { + fields = Arrays.stream(tClass.getDeclaredFields()) + .map(Field::getName) + .filter(n -> !ignoreFields.contains(n)) + .collect(Collectors.toList()); + table.add(reduce(fields)); + } + + /** + * 驼峰转_ + * + * @param str + * @return + */ + private String unCamel(String str) { + // TODO: 2020/3/9 + return null; + } + + /** + * 写comment,包括表名,class名,和外部自定义的comment + */ + private void writeComment() { + table.add(connectCommentPrefix(String.format("tableName:%s", tableName))); + table.add(connectCommentPrefix(String.format("class:%s", tClass.getName()))); + table.add(connectCommentPrefix(String.format("env:%s", DSSWorkFlowConstant.DSS_EXPORT_ENV.getValue()))); + comments.stream().map(this::connectCommentPrefix).forEach(table::add); + } + + /** + * str前加上comment标识 + * + * @param str + * @return + */ + private String connectCommentPrefix(String str) { + return commentPrefix + str; + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/NodeExportService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/NodeExportService.java new file mode 100644 index 000000000..8f8008130 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/NodeExportService.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export; + +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; + +import java.util.List; + +public interface NodeExportService { + void downloadNodeResourceToLocal(String userName, DSSNode dwsNode, String savePath); + void downloadAppConnResourceToLocal(String userName, Long projectId, DSSNode dwsNode, String savePath, Workspace workspace, List dssLabels) throws Exception; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/WorkFlowExportService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/WorkFlowExportService.java new file mode 100644 index 000000000..f93da5d40 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/WorkFlowExportService.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; + +import java.util.List; + +public interface WorkFlowExportService { + + String exportFlowInfo(Long dssProjectId, String projectName, long rootFlowId, String userName, Workspace workspace, List dssLabels) throws Exception; + + void exportFlowResources(String userName, Long projectId, String projectSavePath, String flowJson, String flowName, Workspace workspace,List dssLabels) throws Exception; + + String downloadFlowJsonFromBml(String userName, String resourceId, String version, String savePath); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/MetaExportServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/MetaExportServiceImpl.java new file mode 100644 index 000000000..5e9635fc9 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/MetaExportServiceImpl.java @@ -0,0 +1,78 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export.impl; + +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import com.webank.wedatasphere.dss.workflow.io.export.MetaExportService; +import com.webank.wedatasphere.dss.workflow.io.export.MetaWriter; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Service; +import java.io.*; +import java.util.List; + +@Service +public class MetaExportServiceImpl implements MetaExportService { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + + private final String fileName = "meta.txt"; + + + @Override + public void exportFlowBaseInfo(List allDSSFlows, List allFlowRelations, String savePath) throws IOException { + + try ( + OutputStream outputStream = generateOutputStream(savePath) + ) { + exportFlowBaseInfo(allDSSFlows, outputStream); + exportFlowRelation(allFlowRelations, outputStream); + } + } + + private OutputStream generateOutputStream(String basePath) throws IOException { + return IoUtils.generateExportOutputStream(basePath + File.separator + fileName); + } + + private void exportFlowBaseInfo(List DSSFlows, OutputStream outputStream) throws IOException { + + MetaWriter.of("dss_flow", DSSFlow.class).data(DSSFlows).write(outputStream); + + } + + private InputStream exportFlowBaseInfo(List DSSFlows) throws IOException { + + return MetaWriter.of("dss_flow", DSSFlow.class).data(DSSFlows).write(); + + } + + private void exportFlowRelation(List flowRelations, OutputStream outputStream) throws IOException { + + MetaWriter.of("dss_flow_relation", DSSFlowRelation.class).data(flowRelations).write(outputStream); + + } + + private InputStream exportFlowRelation(List flowRelations) throws IOException { + + return MetaWriter.of("dss_flow_relation", DSSFlowRelation.class).data(flowRelations).write(); + + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/NodeExportServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/NodeExportServiceImpl.java new file mode 100644 index 000000000..325af6dea --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/NodeExportServiceImpl.java @@ -0,0 +1,122 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export.impl; + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.protocol.project.ProjectRelationRequest; +import com.webank.wedatasphere.dss.common.protocol.project.ProjectRelationResponse; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.ref.ExportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.service.RefExportService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RefFactory; +import com.webank.wedatasphere.dss.workflow.dao.NodeInfoMapper; +import com.webank.wedatasphere.dss.workflow.entity.NodeInfo; +import com.webank.wedatasphere.dss.workflow.io.export.NodeExportService; +import com.webank.wedatasphere.dss.workflow.service.BMLService; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.io.File; +import java.util.List; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + + +@Service +public class NodeExportServiceImpl implements NodeExportService { + + private final Logger LOGGER = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private BMLService bmlService; + @Autowired + private NodeInfoMapper nodeInfoMapper; + + private Sender projectSender = DSSSenderServiceFactory.getOrCreateServiceInstance().getProjectServerSender(); + + @Override + public void downloadNodeResourceToLocal(String userName, DSSNode dwsNode, String savePath) { + List resources = dwsNode.getResources(); + if (resources != null) { + resources.forEach(x -> { + // TODO: 2020/6/9 防止前台传的 resources：{{}],后期要去掉 + if (x.getResourceId() != null && x.getFileName() != null && x.getVersion() != null) { + String nodeResourcePath = savePath + File.separator + x.getResourceId() + "_" + x.getVersion() + ".re"; + bmlService.downloadToLocalPath(userName, x.getResourceId(), x.getVersion(), nodeResourcePath); + } else { + LOGGER.warn("Illegal resource information"); + LOGGER.warn("username:{},nodeId:{},nodeName:{},fileName:{},version:{},resourceId:{}", userName, dwsNode.getId(), dwsNode.getName(), x.getFileName(), x.getVersion(), x.getResourceId()); + } + }); + } + } + + @Override + public void downloadAppConnResourceToLocal(String userName, Long projectId, DSSNode dwsNode, String savePath, Workspace workspace,List dssLabels) throws Exception { + NodeInfo nodeInfo = nodeInfoMapper.getWorkflowNodeByType(dwsNode.getNodeType()); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + if (appConn != null) { + DevelopmentIntegrationStandard devStand = ((OnlyDevelopmentAppConn)appConn).getOrCreateDevelopmentStandard(); + + if (null != devStand) { + if (appConn.getAppDesc().getAppInstancesByLabels(dssLabels).size() > 0) { + AppInstance appInstance = appConn.getAppDesc().getAppInstancesByLabels(dssLabels).get(0); + RefExportService refExportService = devStand.getRefExportService(appInstance); + ExportRequestRef requestRef = RefFactory.INSTANCE.newRef(ExportRequestRef.class, refExportService.getClass().getClassLoader(), "com.webank.wedatasphere.dss.appconn." + appConn.getAppDesc().getAppName().toLowerCase()); + //todo request param def + requestRef.setParameter("jobContent", dwsNode.getJobContent()); + requestRef.setParameter("projectId", parseProjectId(projectId, appConn.getAppDesc().getAppName(), dssLabels)); + requestRef.setParameter("nodeType", dwsNode.getNodeType()); + requestRef.setParameter("user", userName); + requestRef.setWorkspace(workspace); + if (null != refExportService) { + Map nodeExportContent = refExportService.getRefExportOperation().exportRef(requestRef).toMap(); + if (nodeExportContent != null) { + String resourceId = nodeExportContent.get("resourceId").toString(); + String version = nodeExportContent.get("version").toString(); + String nodeResourcePath = savePath + File.separator + dwsNode.getId() + ".appconnre"; + bmlService.downloadToLocalPath(userName, resourceId, version, nodeResourcePath); + } else { + LOGGER.error("nodeExportContent is null for projectId {}, dwsNode {}", projectId, dwsNode.getName()); + DSSExceptionUtils.dealErrorException(61023, "nodeExportContent is null", DSSErrorException.class); + } + } + } + }else{ + DSSExceptionUtils.dealErrorException(61024, "Failed to get AppInstance", DSSErrorException.class); + } + } + } + + + private Long parseProjectId(Long dssProjectId, String appconnName, List dssLabels){ + ProjectRelationRequest projectRelationRequest = new ProjectRelationRequest(dssProjectId, appconnName, dssLabels); + ProjectRelationResponse projectRelationResponse = (ProjectRelationResponse) projectSender.ask(projectRelationRequest); + return projectRelationResponse.getAppInstanceProjectId(); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/WorkFlowExportServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/WorkFlowExportServiceImpl.java new file mode 100644 index 000000000..18de36b4f --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/export/impl/WorkFlowExportServiceImpl.java @@ -0,0 +1,199 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.export.impl; + + +import com.webank.wedatasphere.dss.common.entity.IOType; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.common.utils.ZipHelper; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import com.webank.wedatasphere.dss.workflow.common.parser.WorkFlowParser; +import com.webank.wedatasphere.dss.workflow.constant.DSSWorkFlowConstant; +import com.webank.wedatasphere.dss.workflow.dao.FlowMapper; +import com.webank.wedatasphere.dss.workflow.io.export.MetaExportService; +import com.webank.wedatasphere.dss.workflow.io.export.NodeExportService; +import com.webank.wedatasphere.dss.workflow.io.export.WorkFlowExportService; +import com.webank.wedatasphere.dss.workflow.service.BMLService; +import com.webank.wedatasphere.dss.workflow.service.DSSFlowService; +import java.io.File; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + + +@Service +public class WorkFlowExportServiceImpl implements WorkFlowExportService { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private BMLService bmlService; + + @Autowired + private WorkFlowParser workFlowParser; + + @Autowired + private NodeExportService nodeExportService; + + @Autowired + private MetaExportService metaExportService; + + @Autowired + private FlowMapper flowMapper; + + @Autowired + private DSSFlowService flowService; + + + @Override + public String exportFlowInfo(Long dssProjectId, String projectName, long rootFlowId, String userName, Workspace workspace, List dssLabels) throws Exception { + //获取rootFlow,和所有子Flow + DSSFlow rootFlow = flowMapper.selectFlowByID(rootFlowId); + List dssFlowList = new ArrayList<>(); + //生成rootflow及所有子flow + dssFlowList.add(rootFlow); + getAllDssFlowsByRootflowId(rootFlow, dssFlowList); + //生成rootflow及所有子flow的Relations + List flowIds = dssFlowList.stream().map(DSSFlow::getId).collect(Collectors.toList()); + List flowRelations = flowIds.isEmpty() ? new ArrayList<>() : flowMapper.listFlowRelation(flowIds); + String flowExportSaveBasePath = IoUtils.generateIOPath(userName, projectName, ""); + //标记当前导出为project导出 + IoUtils.generateIOType(IOType.FLOW, flowExportSaveBasePath); + //标记当前导出环境 + IoUtils.generateIOEnv(flowExportSaveBasePath); + metaExportService.exportFlowBaseInfo(dssFlowList, flowRelations, flowExportSaveBasePath); + logger.info(userName + "-开始导出Flow：" + rootFlow.getName()); + List dssFlows = new ArrayList<>(); + + for (DSSFlow dssFlow : dssFlowList) { + if (dssFlow.getRootFlow()) { + String savePath = flowExportSaveBasePath + File.separator + dssFlow.getName() + File.separator + dssFlow.getName() + ".json"; + //导出工作流json文件 + String flowJson = downloadFlowJsonFromBml(userName, dssFlow.getResourceId(), dssFlow.getBmlVersion(), savePath); + if (!dssFlow.getHasSaved()) { + logger.info("工作流{}从未保存过，忽略", dssFlow.getName()); + } else if (StringUtils.isNotBlank(flowJson)) { + exportFlowResources(userName, dssProjectId, flowExportSaveBasePath, flowJson, dssFlow.getName(), workspace,dssLabels); + exportAllSubFlows(userName, dssFlow, dssProjectId, flowExportSaveBasePath, workspace,dssLabels); + dssFlows.add(dssFlow); + } else { + String warnMsg = String.format(DSSWorkFlowConstant.PUBLISH_FLOW_REPORT_FORMATE, dssFlow.getName(), dssFlow.getBmlVersion()); + logger.info(warnMsg); + throw new DSSErrorException(90033, warnMsg); + } + } + } + if (dssFlows.isEmpty()) { + throw new DSSErrorException(90037, "该工程没有可以导出的工作流,请检查工作流是否都为空"); + } + //打包导出工程 + return ZipHelper.zipExportProject(flowExportSaveBasePath); + } + + + private void exportAllSubFlows(String userName, DSSFlow dssFlowParent, Long projectId, String projectExportBasePath, Workspace workspace,List dssLabels) throws Exception { + List subFlows = dssFlowParent.getChildren(); + if (subFlows != null) { + for (DSSFlow subFlow : subFlows) { + String savePath = projectExportBasePath + File.separator + subFlow.getName() + File.separator + subFlow.getName() + ".json"; + //导出子flow的json文件 + String flowJson = downloadFlowJsonFromBml(userName, subFlow.getResourceId(), subFlow.getBmlVersion(), savePath); + if (!subFlow.getHasSaved()) { + logger.info("工作流{}从未保存过，忽略", subFlow.getName()); + } else if (StringUtils.isNotBlank(flowJson)) { + exportFlowResources(userName, projectId, projectExportBasePath, flowJson, subFlow.getName(), workspace,dssLabels); + exportAllSubFlows(userName, subFlow, projectId, projectExportBasePath, workspace,dssLabels); + } else { + String warnMsg = String.format(DSSWorkFlowConstant.PUBLISH_FLOW_REPORT_FORMATE, subFlow.getName(), subFlow.getBmlVersion()); + logger.info(warnMsg); + throw new DSSErrorException(90014, warnMsg); + } + } + } + } + + + private String genWorkFlowExportDir(String projectExportPath, String flowName) { + return projectExportPath + File.separator + flowName; + } + + @Override + public void exportFlowResources(String userName, Long projectId, String projectSavePath, String flowJson, String flowName, Workspace workspace,List dssLabels) throws Exception { + String workFlowExportPath = genWorkFlowExportDir(projectSavePath, flowName); + String workFlowResourceSavePath = workFlowExportPath + File.separator + "resource"; + String appConnResourceSavePath = workFlowExportPath + File.separator + "appconn-resource"; + if (StringUtils.isNotEmpty(workFlowExportPath)) { + //导出工作流资源文件 + List resources = workFlowParser.getWorkFlowResources(flowJson); + if (resources != null) { + resources.forEach(resource -> { + downloadFlowResourceFromBml(userName, resource, workFlowResourceSavePath); + }); + } + + //导出工作流节点资源文件,工作流节点appconn文件 + List nodes = workFlowParser.getWorkFlowNodes(flowJson); + if (nodes != null) { + for (DSSNode node : nodes) { + nodeExportService.downloadNodeResourceToLocal(userName, node, workFlowResourceSavePath); + nodeExportService.downloadAppConnResourceToLocal(userName, projectId, node, appConnResourceSavePath, workspace,dssLabels); + } + } + + } else { + throw new DSSErrorException(90067, "工作流导出生成路径为空"); + } + } + + @Override + public String downloadFlowJsonFromBml(String userName, String resourceId, String version, String savePath) { + return bmlService.downloadAndGetFlowJson(userName, resourceId, version, savePath); + } + + private String downloadFlowResourceFromBml(String userName, Resource resource, String savePath) { + String flowResourcePath = savePath + File.separator + resource.getResourceId() + ".re"; + return bmlService.downloadToLocalPath(userName, resource.getResourceId(), resource.getVersion(), flowResourcePath); + } + + private void getAllDssFlowsByRootflowId(DSSFlow parentFlow, List flowList) { + List subFlowIds = flowMapper.selectSavedSubFlowIDByParentFlowID(parentFlow.getId()); + if (subFlowIds.size() > 0) { + for (Long subFlowId : subFlowIds) { + + DSSFlow subDssFlow = flowMapper.selectFlowByID(subFlowId); + if (!subDssFlow.getHasSaved()) { + logger.info("工作流{}从未保存过，忽略", subDssFlow.getName()); + } else { + flowList.add(subDssFlow); + parentFlow.addChildren(subDssFlow); + getAllDssFlowsByRootflowId(subDssFlow, flowList); + } + } + } + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/InputRelationService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/InputRelationService.java new file mode 100644 index 000000000..ba89e3e96 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/InputRelationService.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input; + + +import com.webank.wedatasphere.dss.common.entity.IOEnv; + +public interface InputRelationService { + + boolean projectIsFirstInput(Long sourceProjectID, IOEnv sourceEnv); + + boolean flowIsFirstInput(Long sourceFlowID, IOEnv sourceEnv); + + void insertProjectInputRelation(Long sourceProjectID, IOEnv sourceEnv, Long targetProjectID); + + void insertFlowInputRelation(Long sourceFlowID, IOEnv sourceEnv, Long targetFlowID); + + Long getProjectTargetID(Long sourceProjectID, IOEnv sourceEnv); + + Long getFlowTargetID(Long sourceFlowID, IOEnv sourceEnv); + + void removeFlowInputRelation(IOEnv sourceEnv, Long targetFlowID); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/MetaInputService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/MetaInputService.java new file mode 100644 index 000000000..877b4e475 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/MetaInputService.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input; + + + +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; + +import java.io.IOException; +import java.util.List; + +public interface MetaInputService { + + List inputFlow(String basePath) throws IOException; + + List inputFlowRelation(String basePath) throws IOException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/MetaReader.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/MetaReader.java new file mode 100644 index 000000000..3d14832bf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/MetaReader.java @@ -0,0 +1,165 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input; + +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.lang.reflect.Field; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Locale; +import java.util.stream.Collectors; + +public class MetaReader { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + private final Class tClass; + private final String tableName; + private String commentPrefix = "#"; + private String seperator = "\\|"; + private List datas = new ArrayList<>(); + private List fields = null; + private List comments = new ArrayList<>(); + private List> body = new ArrayList<>(); + private boolean finded = false; + private boolean firstLine = true; + + public MetaReader(Class tClass, String tableName) { + this.tClass = tClass; + this.tableName = tableName; + } + + public static MetaReader of(String tableName, Class tClass) { + return new MetaReader<>(tClass, tableName); + } + + public List read(InputStream inputStream) throws IOException { + try (InputStreamReader streamReader = new InputStreamReader(inputStream); + BufferedReader reader = new BufferedReader(streamReader);) { + readTable(reader); + } + readT(); + return datas; + } + + public String read(InputStream inputStream, String key) throws IOException { + try (InputStreamReader streamReader = new InputStreamReader(inputStream); + BufferedReader reader = new BufferedReader(streamReader);) { + readTable(reader); + } + String comment = comments.stream().filter(c -> c.contains(key + ":")).findFirst().orElse(""); + String[] split = comment.split(":"); + if (split.length > 1) return split[1]; + return ""; + } + + private void readT() { + body.stream().map(DSSExceptionUtils.map(this::lineToT)).forEach(datas::add); + } + + private T lineToT(List list) throws IllegalAccessException, InstantiationException, NoSuchFieldException, ParseException { + T t = tClass.newInstance(); + for (int i = 0; i < list.size(); i++) { + String valueStr = list.get(i); + if ("null".equalsIgnoreCase(valueStr)) continue; + Field declaredField = tClass.getDeclaredField(fields.get(i)); + declaredField.setAccessible(true); + Object value = null; + String type = declaredField.getType().getSimpleName(); + switch (type) { + case "String": + value = valueStr; + break; + case "Date": + value = new SimpleDateFormat("EEE MMM dd HH:mm:ss ZZZ yyyy", Locale.ENGLISH).parse(valueStr); + break; + case "Long": + value = Long.valueOf(valueStr); + break; + case "Boolean": + value = Boolean.valueOf(valueStr); + break; + case "Integer": + value = Integer.valueOf(valueStr); + break; + default: + logger.warn(String.format("unsupport type %s", type)); + } + declaredField.set(t, value); + } + return t; + } + + private void readTable(BufferedReader reader) throws IOException { + String line = null; + while ((line = reader.readLine()) != null) { + if (!finded && !shut(line)) { + continue; + } + if (finded && isTableName(line)) { + break; + } + if (shut(line)) { + finded = true; + continue; + } + // TODO: 2020/3/9 + if (isComment(line)) { + comments.add(line); + continue; + } + if (firstLine) { + //handle head + fields = Arrays.stream(line.split(seperator)).collect(Collectors.toList()); + firstLine = false; + continue; + } + body.add(Arrays.stream(line.split(seperator)).collect(Collectors.toList())); + //handle body + } + } + + private boolean isComment(String str) { + return str.startsWith(commentPrefix); + } + + private boolean isTableName(String str) { + return isComment(str) && str.contains(commentPrefix + "tableName:"); + } + + private boolean isClass(String str) { + return isComment(str) && str.contains(commentPrefix + "class:"); + } + + private String getComment(String str) { + return str.substring(1); + } + + private boolean shut(String str) { + return isTableName(str) && getComment(str).equals(String.format("tableName:%s", tableName)); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/NodeInputService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/NodeInputService.java new file mode 100644 index 000000000..4e2d95fc3 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/NodeInputService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input; + + +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; + +import java.io.IOException; + +public interface NodeInputService { + String uploadResourceToBml(String userName, String nodeJson, String inputResourcePath, String projectName) throws IOException; + String uploadAppConnResource(String userName, String projectName, DSSFlow DSSFlow, String nodeJson, String flowContextId, String appConnResourcePath, Workspace workspace, String orcVerson) throws IOException; + String updateNodeSubflowID(String nodeJson, long subflowID) throws IOException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/WorkFlowInputService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/WorkFlowInputService.java new file mode 100644 index 000000000..da8875889 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/WorkFlowInputService.java @@ -0,0 +1,56 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input; + + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; + +import java.io.IOException; +import java.util.List; + +public interface WorkFlowInputService { + + /** + * + * @param userName + * @param dssFlow + * @param projectName + * @param inputProjectPath + * @param parentFlowId + * @param workspace + * @param orcVersion + * @throws DSSErrorException + + * @throws IOException + */ + void inputWorkFlow(String userName, String workspaceName, DSSFlow dssFlow, String version, String projectName, String inputProjectPath, Long parentFlowId, Workspace workspace, String orcVersion,String contextId) throws DSSErrorException,IOException; + + /** + * save flow to db + * @param projectId + * @param userName + * @param dssFlows + * @param DSSFlowRelations + * @param sourceEnv + * @return + */ + List persistenceFlow(Long projectId, String userName, List dssFlows, List DSSFlowRelations, IOEnv sourceEnv); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/InputRelationServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/InputRelationServiceImpl.java new file mode 100644 index 000000000..c92386869 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/InputRelationServiceImpl.java @@ -0,0 +1,86 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input.impl; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.common.entity.IOType; +import com.webank.wedatasphere.dss.common.entity.InputRelation; +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.workflow.dao.InputRelationMapper; +import com.webank.wedatasphere.dss.workflow.io.input.InputRelationService; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +@Service +public class InputRelationServiceImpl implements InputRelationService { + + @Autowired + private InputRelationMapper inputRelationMapper; + + @Override + public boolean projectIsFirstInput(Long sourceProjectID, IOEnv sourceEnv) { + InputRelation inputRelation = inputRelationMapper.selectInputRelation(IOType.PROJECT.name(), sourceEnv, sourceProjectID, IoUtils.getDSSServerEnv()); + return inputRelation == null; + } + + @Override + public boolean flowIsFirstInput(Long sourceFlowID, IOEnv sourceEnv) { + InputRelation inputRelation = inputRelationMapper.selectInputRelation(IOType.FLOW.name(), sourceEnv, sourceFlowID, IoUtils.getDSSServerEnv()); + return inputRelation == null; + } + + @Override + public void insertProjectInputRelation(Long sourceProjectID, IOEnv sourceEnv, Long targetProjectID) { + InputRelation inputRelation = buildInputRelation(sourceProjectID, sourceEnv, targetProjectID, IOType.PROJECT.name()); + inputRelationMapper.insertInputRelation(inputRelation); + } + + @Override + public void insertFlowInputRelation(Long sourceFlowID, IOEnv sourceEnv, Long targetFlowID) { + InputRelation inputRelation = buildInputRelation(sourceFlowID, sourceEnv, targetFlowID, IOType.FLOW.name()); + inputRelationMapper.insertInputRelation(inputRelation); + } + + @Override + public Long getProjectTargetID(Long sourceProjectID, IOEnv sourceEnv) { + return inputRelationMapper.selectInputRelation(IOType.PROJECT.name(),sourceEnv,sourceProjectID,IoUtils.getDSSServerEnv()).getTargetID(); + } + + @Override + public Long getFlowTargetID(Long sourceFlowID, IOEnv sourceEnv) { + InputRelation inputRelation = inputRelationMapper.selectInputRelation(IOType.FLOW.name(), sourceEnv, sourceFlowID, IoUtils.getDSSServerEnv()); + if(inputRelation == null) return null ;//新增的子flowID查询会为null + return inputRelation.getTargetID(); + } + + @Override + public void removeFlowInputRelation(IOEnv sourceEnv, Long targetFlowID) { + inputRelationMapper.removeInputRelation(IOType.FLOW.name(),sourceEnv,targetFlowID,IoUtils.getDSSServerEnv()); + } + + public InputRelation buildInputRelation(Long sourceID, IOEnv sourceEnv, Long targetID, String type) { + InputRelation inputRelation = new InputRelation(); + inputRelation.setSourceEnv(sourceEnv); + inputRelation.setTargetEnv(IoUtils.getDSSServerEnv()); + inputRelation.setSourceID(sourceID); + inputRelation.setTargetID(targetID); + inputRelation.setType(type); + return inputRelation; + } + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/MetaInputServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/MetaInputServiceImpl.java new file mode 100644 index 000000000..03b4ca835 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/MetaInputServiceImpl.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input.impl; + +import com.webank.wedatasphere.dss.common.utils.IoUtils; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import com.webank.wedatasphere.dss.workflow.io.input.MetaInputService; +import com.webank.wedatasphere.dss.workflow.io.input.MetaReader; +import org.springframework.stereotype.Service; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.InputStream; +import java.util.List; + + +@Service +public class MetaInputServiceImpl implements MetaInputService { + + // TODO: 2020/3/13 防止表结构发生改变的version 字段的添加 + + private final String fileName = "meta.txt"; + + + @Override + public List inputFlow(String basePath) throws IOException { + try (InputStream inputStream = generateInputstream(basePath)) { + return MetaReader.of("dss_flow", DSSFlow.class).read(inputStream); + } + } + + @Override + public List inputFlowRelation(String basePath) throws IOException { + try (InputStream inputStream = generateInputstream(basePath)) { + return MetaReader.of("dss_flow_relation", DSSFlowRelation.class).read(inputStream); + } + } + + /** + * 获取inputStream + * + * @param basePath + * @return + * @throws FileNotFoundException + */ + private InputStream generateInputstream(String basePath) throws IOException { + return IoUtils.generateInputInputStream(basePath + File.separator + fileName); + } + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/NodeInputServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/NodeInputServiceImpl.java new file mode 100644 index 000000000..28ccf692d --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/NodeInputServiceImpl.java @@ -0,0 +1,138 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input.impl; + + +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.contextservice.service.impl.ContextServiceImpl; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.parser.NodeParser; +import com.webank.wedatasphere.dss.workflow.entity.CommonAppConnNode; +import com.webank.wedatasphere.dss.workflow.io.input.NodeInputService; +import com.webank.wedatasphere.dss.workflow.service.BMLService; +import com.webank.wedatasphere.dss.workflow.service.WorkflowNodeService; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.util.*; + +@Service +public class NodeInputServiceImpl implements NodeInputService { + @Autowired + private BMLService bmlService; + + @Autowired + private NodeParser nodeParser; + + @Autowired + private WorkflowNodeService nodeService; + + private static ContextService contextService = ContextServiceImpl.getInstance(); + + private final Logger logger = LoggerFactory.getLogger(this.getClass()); + + + @Override + public String uploadResourceToBml(String userName, String nodeJson, String inputResourcePath, String projectName) throws IOException { + List resources = nodeParser.getNodeResource(nodeJson); + if (resources != null && resources.size() > 0) { + resources.forEach(resource -> { + if (resource.getVersion() != null && resource.getFileName() != null && resource.getResourceId() != null) { + InputStream resourceInputStream = readResource(userName, resource, inputResourcePath); + Map bmlReturnMap = bmlService.upload(userName, + resourceInputStream, UUID.randomUUID().toString() + ".json", projectName); + resource.setResourceId(bmlReturnMap.get("resourceId").toString()); + resource.setVersion(bmlReturnMap.get("version").toString()); + } else { + logger.warn("Illegal resource information"); + logger.warn("username:{},fileName:{},version:{},resourceId:{}", userName, resource.getFileName(), resource.getVersion(), resource.getResourceId()); + } + }); + } + return nodeParser.updateNodeResource(nodeJson, resources); + } + + private InputStream readResource(String userName, Resource resource, String flowResourcePath) { + String readPath = flowResourcePath + resource.getResourceId() + "_" + resource.getVersion() + ".re"; + return bmlService.readLocalResourceFile(userName, readPath); + } + + @Override + public String uploadAppConnResource(String userName, String projectName, DSSFlow dssFlow, String nodeJson, String flowContextId, String appConnResourcePath, Workspace workspace, String orcVersion) throws IOException { + Map nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(nodeJson, Map.class); + String nodeType = nodeJsonMap.get("jobType").toString(); + + + String nodeId = nodeJsonMap.get("id").toString(); + + Map nodeContent = (LinkedHashMap) nodeJsonMap.get("jobContent"); + CommonAppConnNode appConnNode = new CommonAppConnNode(); + appConnNode.setId(nodeId); + appConnNode.setNodeType(nodeType); + appConnNode.setJobContent(nodeContent); + appConnNode.setFlowId(dssFlow.getId()); + appConnNode.setProjectId(dssFlow.getProjectID()); + + Map nodeExportContent = null; + + if (nodeService != null) { + logger.info("appConn NodeService is exist"); + String nodeResourcePath = appConnResourcePath + File.separator + nodeId + ".appconnre"; + File file = new File(nodeResourcePath); + if (file.exists()) { + InputStream resourceInputStream = bmlService.readLocalResourceFile(userName, nodeResourcePath); + Map bmlReturnMap = bmlService.upload(userName, resourceInputStream, UUID.randomUUID().toString() + ".json", + projectName); + try { + nodeExportContent = nodeService.importNode(userName, appConnNode, bmlReturnMap, workspace, orcVersion); + } catch (Exception e) { + logger.error("failed to import node ", e); + } + if (nodeExportContent != null) { + if (nodeExportContent.get("project_id") != null) { + Long newProjectId = Long.parseLong(nodeExportContent.get("project_id").toString()); + logger.warn(String.format("new appConn node add into dss,dssProjectId: %s,newProjectId: %s", appConnNode.getProjectId(), newProjectId)); + nodeExportContent.remove("project_id"); + } + nodeJsonMap.replace("jobContent", nodeExportContent); + appConnNode.setJobContent(nodeExportContent); + return BDPJettyServerHelper.jacksonJson().writeValueAsString(nodeJsonMap); + } + }else{ + logger.error("appConn node resource file does not exists."+nodeId); + } + } + + + return nodeJson; + } + + @Override + public String updateNodeSubflowID(String nodeJson, long subflowID) throws IOException { + + return nodeParser.updateSubFlowID(nodeJson, subflowID); + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/WorkFlowInputServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/WorkFlowInputServiceImpl.java new file mode 100644 index 000000000..b17b2abc4 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/input/impl/WorkFlowInputServiceImpl.java @@ -0,0 +1,269 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.input.impl; + +import com.webank.wedatasphere.dss.common.entity.IOEnv; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.contextservice.service.impl.ContextServiceImpl; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import com.webank.wedatasphere.dss.workflow.common.parser.NodeParser; +import com.webank.wedatasphere.dss.workflow.common.parser.WorkFlowParser; +import com.webank.wedatasphere.dss.workflow.dao.FlowMapper; +import com.webank.wedatasphere.dss.workflow.io.input.InputRelationService; +import com.webank.wedatasphere.dss.workflow.io.input.NodeInputService; +import com.webank.wedatasphere.dss.workflow.io.input.WorkFlowInputService; +import com.webank.wedatasphere.dss.workflow.service.BMLService; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.BeanUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.util.*; +import java.util.stream.Collectors; + +@Service +public class WorkFlowInputServiceImpl implements WorkFlowInputService { + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private BMLService bmlService; + + @Autowired + private WorkFlowParser workFlowParser; + + @Autowired + private NodeInputService nodeInputService; + + @Autowired + private NodeParser nodeParser; + + @Autowired + private InputRelationService inputRelationService; + + @Autowired + private FlowMapper flowMapper; + private static ContextService contextService = ContextServiceImpl.getInstance(); + + @Override + public void inputWorkFlow(String userName, + String workspaceName, + DSSFlow dssFlow, + String version, + String projectName, + String inputProjectPath, + Long parentFlowId, + Workspace workspace, + String orcVersion, + String contextId) throws DSSErrorException, IOException { + //todo 不同服务共享导入工程文件包,可采用共享存储 + String flowInputPath = inputProjectPath + File.separator + dssFlow.getName(); + String flowJsonPath = flowInputPath + File.separator + dssFlow.getName() + ".json"; + String creator = dssFlow.getCreator(); + String flowJson = bmlService.readLocalFlowJsonFile(userName, flowJsonPath); + //如果包含subflow,需要一同导入subflow内容，并更新parrentflow的json内容 + // TODO: 2020/7/31 优化update方法里面的saveContent + String updateFlowJson = updateFlowContextId(flowJson, contextId); + updateFlowJson = inputWorkFlowNodes(userName, projectName, updateFlowJson, dssFlow, flowInputPath, workspace, orcVersion); + List subFlows = dssFlow.getChildren(); + if (subFlows != null) { + for (DSSFlow subFlow : subFlows) { + inputWorkFlow(userName, workspaceName, subFlow, version, projectName, inputProjectPath, dssFlow.getId(), workspace, orcVersion,contextId); + } + } + + updateFlowJson = uploadFlowResourceToBml(userName, updateFlowJson, flowInputPath, projectName); + + DSSFlow updateDssFlow = uploadFlowJsonToBml(userName, projectName, dssFlow, updateFlowJson); + //todo add dssflow to database + contextService.checkAndSaveContext(updateFlowJson, String.valueOf(parentFlowId)); + flowMapper.updateFlowInputInfo(updateDssFlow); + + + } + + private String updateFlowContextId(String flowJson, String contextId) throws IOException { + +// String contextID = contextService.checkAndInitContext(flowJson, parentFlowIdStr, workspace, projectName, flowName, flowVersion, userName); + String updatedFlowJson = workFlowParser.updateFlowJsonWithKey(flowJson, "contextID", contextId); + return updatedFlowJson; + } + + private String inputWorkFlowNodes(String userName, String projectName, String flowJson, DSSFlow dssFlow, String flowPath, Workspace workspace, String orcVersion) throws DSSErrorException, IOException { + List nodeJsonList = workFlowParser.getWorkFlowNodesJson(flowJson); + if (nodeJsonList == null) { + throw new DSSErrorException(90073, "工作流内没有工作流节点，导入失败" + dssFlow.getName()); + } + String updateContextId = workFlowParser.getValueWithKey(flowJson, "contextID"); + if (nodeJsonList.size() == 0) { + return flowJson; + } + List subflows = (List) dssFlow.getChildren(); + String workFlowResourceSavePath = flowPath + File.separator + "resource" + File.separator; + String appConnResourceSavePath = flowPath + File.separator + "appconn-resource"; + List> nodeJsonListRes = new ArrayList<>(); + if (nodeJsonList.size() > 0) { + for (String nodeJson : nodeJsonList) { + // TODO: 2020/3/20 暂时注视掉appconn相关 + String updateNodeJson = nodeInputService.uploadResourceToBml(userName, nodeJson, workFlowResourceSavePath, projectName); + updateNodeJson = nodeInputService.uploadAppConnResource(userName, projectName, + dssFlow, updateNodeJson, updateContextId, appConnResourceSavePath, workspace, orcVersion); + + Map nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(updateNodeJson, Map.class); + //更新subflowID + String nodeType = nodeJsonMap.get("jobType").toString(); + if ("workflow.subflow".equals(nodeType)) { + String subFlowName = nodeJsonMap.get("title").toString(); + List DSSFlowList = subflows.stream().filter(subflow -> + subflow.getName().equals(subFlowName) + ).collect(Collectors.toList()); + if (DSSFlowList.size() == 1) { + updateNodeJson = nodeInputService.updateNodeSubflowID(updateNodeJson, DSSFlowList.get(0).getId()); + nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(updateNodeJson, Map.class); + nodeJsonListRes.add(nodeJsonMap); + } else if (DSSFlowList.size() > 1) { + logger.error("工程内存在重复的子工作流节点名称，导入失败" + subFlowName); + throw new DSSErrorException(90077, "工程内存在重复的子工作流节点名称，导入失败" + subFlowName); + } else { + logger.error("工程内存在重复的子工作流节点名称，导入失败" + subFlowName); + throw new DSSErrorException(90078, "工程内未能找到子工作流节点，导入失败" + subFlowName); + } + } else { + nodeJsonListRes.add(nodeJsonMap); + } + } + } + + return workFlowParser.updateFlowJsonWithKey(flowJson, "nodes", nodeJsonListRes); + + } + + private String uploadFlowResourceToBml(String userName, String flowJson, String flowResourcePath, String projectName) throws IOException { + + List resourceList = workFlowParser.getWorkFlowResources(flowJson); + //上传文件获取resourceId和version save应该是已经有 + if (resourceList != null) { + resourceList.forEach(resource -> { + InputStream resourceInputStream = readFlowResource(userName, resource, flowResourcePath); + Map bmlReturnMap = bmlService.upload(userName, resourceInputStream, UUID.randomUUID().toString() + ".json", projectName); + resource.setResourceId(bmlReturnMap.get("resourceId").toString()); + resource.setVersion(bmlReturnMap.get("version").toString()); + }); + if (resourceList.size() == 0) { + return flowJson; + } + } + return workFlowParser.updateFlowJsonWithKey(flowJson, "resources", resourceList); + } + + private InputStream readFlowResource(String userName, Resource resource, String flowResourcePath) { + // TODO: 2020/3/20 和导出统一,资源都放resouce 如有问题,后再修改 + String readPath = flowResourcePath + File.separator + "resource" + File.separator + resource.getResourceId() + ".re"; + return bmlService.readLocalResourceFile(userName, readPath); + } + + + public DSSFlow uploadFlowJsonToBml(String userName, String projectName, DSSFlow dssFlow, String flowJson) { + //获取rsourceId，就是jsonPath + Long flowID = dssFlow.getId(); + String resourceId = dssFlow.getResourceId(); + //上传文件获取resourceId和version save应该是已经有 + Map bmlReturnMap; +// if (resourceId != null) { +// bmlReturnMap = bmlService.update(userName, resourceId, flowJson); +// } else { + //上传文件获取resourceId和version save应该是已经有 + bmlReturnMap = bmlService.upload(userName, flowJson, UUID.randomUUID().toString() + ".json", projectName); +// } + + dssFlow.setCreator(userName); + dssFlow.setBmlVersion(bmlReturnMap.get("version").toString()); + dssFlow.setResourceId(bmlReturnMap.get("resourceId").toString()); + dssFlow.setDescription("import update workflow"); + dssFlow.setSource("导入更新"); + + //version表中插入数据 + return dssFlow; + } + + /** + * @param projectId 新环境的projectID + * @param userName + * @param dssFlows + * @param sourceEnv + * @return + */ + @Override + public List persistenceFlow(Long projectId, String userName, List dssFlows, + List dssFlowRelations, IOEnv sourceEnv) { + List rootFlows = dssFlows.stream().filter(DSSFlow::getRootFlow).collect(Collectors.toList()); + return rootFlows.stream().map(rf -> setSubFlow(rf, dssFlows, dssFlowRelations, sourceEnv, projectId, userName, null)) + .collect(Collectors.toList()); + } + + public DSSFlow setSubFlow(DSSFlow dssFlow, List dssFlows, + List dssFlowRelations, + IOEnv sourceEnv, + Long projectId, + String username, DSSFlow parentFlow) { + DSSFlow cyFlow = new DSSFlow(); + BeanUtils.copyProperties(dssFlow, cyFlow, "children", "flowVersions"); + //封装flow信息 + cyFlow.setProjectID(projectId); + cyFlow.setCreator(username); + cyFlow.setCreateTime(new Date()); + cyFlow.setId(null); + flowMapper.insertFlow(cyFlow); + //插入input 关联信息 + + //inputRelationService.insertFlowInputRelation(dssFlow.getId(), sourceEnv, cyFlow.getId()); + + //插入或者更新relation表 + if (parentFlow != null) { + persistenceFlowRelation(cyFlow.getId(), parentFlow.getId()); + if (parentFlow.getChildren() == null) { + parentFlow.setChildren(new ArrayList()); + } + parentFlow.addChildren(cyFlow); + } + List subFlowIds = dssFlowRelations.stream().filter(r -> r.getParentFlowID().equals(dssFlow.getId())).map(DSSFlowRelation::getFlowID).collect(Collectors.toList()); + for (Long subFlowId : subFlowIds) { + DSSFlow subDSSFlow = dssFlows.stream().filter(f -> subFlowId.equals(f.getId())).findFirst().orElse(null); + if (dssFlow.getChildren() == null) { + dssFlow.setChildren(new ArrayList()); + } + dssFlow.addChildren(subDSSFlow); + setSubFlow(subDSSFlow, dssFlows, dssFlowRelations, sourceEnv, projectId, username, cyFlow); + } + return cyFlow; + } + + private void persistenceFlowRelation(Long flowID, Long parentFlowID) { + DSSFlowRelation relation = flowMapper.selectFlowRelation(flowID, parentFlowID); + if (relation == null) { + flowMapper.insertFlowRelation(flowID, parentFlowID); + } + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/IOJobHook.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/IOJobHook.java new file mode 100644 index 000000000..3fd24c50b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/IOJobHook.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.scheduler; + +import com.webank.wedatasphere.dss.workflow.scheduler.DssJob; +import com.webank.wedatasphere.dss.workflow.scheduler.DssJobHook; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class IOJobHook implements DssJobHook { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Override + public void preExecute(DssJob job) { + + + } + + @Override + public void postExecute(DssJob job) { + + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/IOManager.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/IOManager.java new file mode 100644 index 000000000..22dd7a715 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/IOManager.java @@ -0,0 +1,42 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.scheduler; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.workflow.scheduler.DssJobManager; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; + +@Component +public class IOManager extends DssJobManager { + + @PostConstruct + public void init() { + addDssJobHook(new IOJobHook()); + } + + public void submitIoJob(String userName, String comment, Long... flowIDs) throws DSSErrorException { + PublishJob exportJob = new PublishJob(); + exportJob.setDssJobListener(this); + exportJob.setUser(userName); + exportJob.setFlowIDs(flowIDs); + exportJob.setComment(comment); + submit(exportJob); + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/PublishJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/PublishJob.java new file mode 100644 index 000000000..86173edd7 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/io/scheduler/PublishJob.java @@ -0,0 +1,64 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.io.scheduler; + +import com.webank.wedatasphere.dss.workflow.scheduler.DssJob; + +public class PublishJob extends DssJob { + + + + private String comment; + + private Long[] flowIDs; + + + @Override + public void run() { + try { + dssJobListener.onJobRunning(this, "导出job正在运行：" + id); + //todo 工作流发布逻辑 + + dssJobListener.onJobSucceed(this, "导出job执行成功：" + id); + } catch (Throwable e) { + logger.error("导出job执行失败，原因：", e); + Throwable cause = e.getCause(); + String errorMsg = cause == null ? e.getMessage() : cause.getMessage(); + dssJobListener.onJobFailed(this, "导出job执行失败：" + errorMsg); + } + } + + + + public Long[] getFlowIDs() { + return flowIDs; + } + + public void setFlowIDs(Long[] flowIDs) { + this.flowIDs = flowIDs; + } + + public String getComment() { + return comment; + } + + public void setComment(String comment) { + this.comment = comment; + } + + +} \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/Lock.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/lock/Lock.java similarity index 87% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/Lock.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/lock/Lock.java index 0e381b2ec..a6b4bb802 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/Lock.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/lock/Lock.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,14 +14,13 @@ * */ -package com.webank.wedatasphere.dss.server.lock; +package com.webank.wedatasphere.dss.workflow.lock; import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; - @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD}) public @interface Lock { diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/LockEnum.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/lock/LockEnum.java similarity index 83% rename from dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/LockEnum.java rename to dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/lock/LockEnum.java index 3b532d460..2dc310f10 100644 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/LockEnum.java +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/lock/LockEnum.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,8 +14,7 @@ * */ -package com.webank.wedatasphere.dss.server.lock; - +package com.webank.wedatasphere.dss.workflow.lock; public interface LockEnum { String ADD = "+"; diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/parser/DefaultNodeParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/parser/DefaultNodeParser.java new file mode 100644 index 000000000..2683c58d1 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/parser/DefaultNodeParser.java @@ -0,0 +1,84 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.parser; + +import com.google.gson.Gson; +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.workflow.common.parser.NodeParser; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.apache.commons.lang.StringUtils; +import org.springframework.stereotype.Component; + +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +@Component +public class DefaultNodeParser implements NodeParser { + @Override + public String updateNodeResource(String nodeJson, List resources) throws IOException { + if(resources.size() ==0){ + return nodeJson; + } + Map nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(nodeJson, Map.class); + nodeJsonMap.replace("resources",resources); + + return BDPJettyServerHelper.jacksonJson().writeValueAsString(nodeJsonMap); + } + + @Override + public String updateNodeJobContent(String nodeJson, Map nodeExportContent) throws IOException { + if(nodeExportContent ==null){ + return nodeJson; + } + Map nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(nodeJson, Map.class); + nodeJsonMap.replace("jobContent",nodeExportContent); + return BDPJettyServerHelper.jacksonJson().writeValueAsString(nodeJsonMap); + } + + @Override + public String updateSubFlowID(String nodeJson, long subflowId) throws IOException { + Map nodeContentMap = new HashMap<>(); + nodeContentMap.put("embeddedFlowId",subflowId); + return updateNodeJobContent(nodeJson,nodeContentMap); + } + + @Override + public String getNodeValue(String key, String nodeJson) throws IOException { + if(StringUtils.isEmpty(nodeJson)){ + return null; + } + Map nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(nodeJson, Map.class); + return nodeJsonMap.get(key).toString(); + + } + + @Override + public List getNodeResource(String nodeJson) { + Gson gson = new Gson(); + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(nodeJson).getAsJsonObject(); + JsonArray resourcesJsonArray = jsonObject.getAsJsonArray("resources"); + List resources = gson.fromJson(resourcesJsonArray, new com.google.gson.reflect.TypeToken>() { + }.getType()); + return resources; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/parser/DefaultWorkFlowParser.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/parser/DefaultWorkFlowParser.java new file mode 100644 index 000000000..01c057438 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/parser/DefaultWorkFlowParser.java @@ -0,0 +1,98 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.parser; + +import com.google.common.reflect.TypeToken; +import com.google.gson.Gson; +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.webank.wedatasphere.dss.common.entity.Resource; +import com.webank.wedatasphere.dss.common.entity.node.DSSNode; +import com.webank.wedatasphere.dss.common.entity.node.DSSNodeDefault; +import com.webank.wedatasphere.dss.workflow.common.parser.WorkFlowParser; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.springframework.stereotype.Component; + +import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +@Component +public class DefaultWorkFlowParser implements WorkFlowParser { + @Override + public List getWorkFlowResources(String workFlowJson) { + Gson gson = new Gson(); + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(workFlowJson).getAsJsonObject(); + JsonArray resourcesJsonArray = jsonObject.getAsJsonArray("resources"); + List resources = gson.fromJson(resourcesJsonArray, new com.google.gson.reflect.TypeToken>() { + }.getType()); + return resources; + } + + @Override + public List getWorkFlowNodes(String workFlowJson) { + Gson gson = new Gson(); + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(workFlowJson).getAsJsonObject(); + JsonArray nodeJsonArray = jsonObject.getAsJsonArray("nodes"); + List dwsNodes = gson.fromJson(nodeJsonArray, new TypeToken>() { + }.getType()); + return dwsNodes; + } + + @Override + public List getWorkFlowNodesJson(String workFlowJson) { + Gson gson = new Gson(); + JsonParser parser = new JsonParser(); + JsonObject jsonObject = parser.parse(workFlowJson).getAsJsonObject(); + JsonArray nodeJsonArray = jsonObject.getAsJsonArray("nodes"); + if(nodeJsonArray==null){ + return null; + } + List nodeJsonList = gson.fromJson(nodeJsonArray.toString(), new TypeToken>() { + }.getType()); + return nodeJsonList.stream().map(gson::toJson).collect(Collectors.toList()); + } + + @Override + public String updateFlowJsonWithKey(String workFlowJson, String key, Object value) throws IOException { + if(value == null || key == null){ + return workFlowJson; + } + Map flowJsonObject = BDPJettyServerHelper.jacksonJson().readValue(workFlowJson, Map.class); + + + flowJsonObject.replace(key,value); + String updatedJson = BDPJettyServerHelper.jacksonJson().writeValueAsString(flowJsonObject); + return updatedJson; + } + + @Override + public String getValueWithKey(String workFlowJson, String key) throws IOException { + if(key == null){ + return null; + } + Map flowJsonObject = BDPJettyServerHelper.jacksonJson().readValue(workFlowJson, Map.class); + + + return flowJsonObject.get(key).toString(); + + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/ContextServiceRestful.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/ContextServiceRestful.java new file mode 100644 index 000000000..aa032fccf --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/ContextServiceRestful.java @@ -0,0 +1,64 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.restful; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.workflow.cs.service.CSTableService; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import org.codehaus.jackson.JsonNode; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.Consumes; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; + +@Component +@Path("/dss/workflow") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class ContextServiceRestful { + + @Autowired + private CSTableService csTableService; + + @POST + @Path("tables") + public Response tables(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { + String userName = SecurityFilter.getLoginUsername(req); + String contextIDStr = json.get("contextID").getTextValue(); + String nodeName = json.get("nodeName").getTextValue(); + return Message.messageToResponse(Message.ok().data("tables", csTableService.queryTables("default", contextIDStr, nodeName))); + } + + + @POST + @Path("columns") + public Response queryTableMeta(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { + String userName = SecurityFilter.getLoginUsername(req); + String contextIDStr = json.get("contextID").getTextValue(); + String contextKeyStr = json.get("contextKey").getTextValue(); + return Message.messageToResponse(Message.ok().data("columns", csTableService.queryTableMeta("default", contextIDStr, contextKeyStr))); + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/FlowRestfulApi.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/FlowRestfulApi.java new file mode 100644 index 000000000..661a3a81a --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/FlowRestfulApi.java @@ -0,0 +1,296 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.restful; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.label.LabelKeyConvertor; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.contextservice.service.impl.ContextServiceImpl; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseConvertOrchestrator; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.dss.workflow.WorkFlowManager; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.constant.DSSWorkFlowConstant; +import com.webank.wedatasphere.dss.workflow.service.DSSFlowService; +import com.webank.wedatasphere.dss.workflow.service.PublishService; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import java.io.IOException; +import java.util.*; +import java.util.Map.Entry; +import java.util.stream.Collectors; +import java.util.stream.StreamSupport; +import javax.servlet.http.HttpServletRequest; +import javax.validation.constraints.NotNull; +import javax.ws.rs.Consumes; +import javax.ws.rs.GET; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.QueryParam; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import org.apache.commons.lang.StringUtils; +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.map.ObjectMapper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +@Component +@Path("/dss/workflow") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class FlowRestfulApi { + + @Autowired + private DSSFlowService flowService; + + + private ContextService contextService = ContextServiceImpl.getInstance(); + + @Autowired + private PublishService publishService; + + @Autowired + private WorkFlowManager workFlowManager; + + ObjectMapper mapper = new ObjectMapper(); + + + private static final Logger LOGGER = LoggerFactory.getLogger(FlowRestfulApi.class); + + @POST + @Path("/addFlow") +// @ProjectPrivChecker + public Response addFlow(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException, JsonProcessingException { + //如果是子工作流，那么分类应该是和父类一起的？ + String userName = SecurityFilter.getLoginUsername(req); + // TODO: 2019/5/23 flowName工程名下唯一校验 + String name = json.get("name").getTextValue(); + String workspaceName = json.get("workspaceName").getTextValue(); + String projectName = json.get("projectName").getTextValue(); + String version = json.get("version").getTextValue(); + String description = json.get("description") == null ? null : json.get("description").getTextValue(); + Long parentFlowID = json.get("parentFlowID") == null ? null : json.get("parentFlowID").getLongValue(); + String uses = json.get("uses") == null ? null : json.get("uses").getTextValue(); + JsonNode dssLabelsJsonNode = json.get(DSSCommonUtils.DSS_LABELS_KEY); + List dssLabelList = new ArrayList<>(); + if (dssLabelsJsonNode != null && dssLabelsJsonNode.getElements().hasNext()) { + Iterator nodeList = dssLabelsJsonNode.getElements(); + while (nodeList.hasNext()) { + JsonNode objNode =nodeList.next(); + EnvDSSLabel dssLabel = new EnvDSSLabel(objNode.getTextValue()); + dssLabelList.add(dssLabel); + } + } + String contextId = contextService.createContextID(workspaceName, projectName, name, version, userName); + + DSSFlow dssFlow = workFlowManager.createWorkflow(userName,name,contextId,description,parentFlowID,uses,null,dssLabelList); + + // TODO: 2019/5/16 空值校验，重复名校验 + return Message.messageToResponse(Message.ok().data("flow", dssFlow)); + } + + + @POST + @Path("/publishWorkflow") + public Response publishWorkflow(@Context HttpServletRequest request, JsonNode jsonNode) { + Long workflowId = jsonNode.get("workflowId").getLongValue(); +// Map labels = StreamSupport.stream(Spliterators.spliteratorUnknownSize(dssLabel.getFields(), +// Spliterator.ORDERED), false).collect(Collectors.toMap(Entry::getKey, entry -> entry.getValue().getTextValue())); + //todo modify by front label + JsonNode labelJsonNode = jsonNode.get(DSSCommonUtils.DSS_LABELS_KEY); + String dssLabel = labelJsonNode.get(LabelKeyConvertor.ROUTE_LABEL_KEY).getTextValue(); + Map labels=new HashMap<>(); + labels.put(EnvDSSLabel.DSS_ENV_LABEL_KEY,dssLabel); + String comment = jsonNode.get("comment").getTextValue(); + Workspace workspace = SSOHelper.getWorkspace(request); + String publishUser = SecurityFilter.getLoginUsername(request); + Message message; + try{ + String taskId = publishService.submitPublish(publishUser, workflowId, labels, workspace, comment); + LOGGER.info("submit publish task ok ,taskId is {}.", taskId); + if (StringUtils.isNotEmpty(taskId)){ + message = Message.ok("生成工作流发布任务成功").data("releaseTaskId", taskId); + } else{ + LOGGER.error("taskId {} is error.", taskId); + message = Message.error("发布工作流失败"); + } + }catch(final Throwable t){ + LOGGER.error("failed to submit publish task for workflow id {}.", workflowId, t); + message = Message.error("发布工作流失败"); + } + return Message.messageToResponse(message); + } + + + /** + * 获取发布任务状态 + * @param request + * @param releaseTaskId + * @return + */ + + @GET + @Path("/getReleaseStatus") + public Response getReleaseStatus(@Context HttpServletRequest request, + @NotNull(message = "查询的发布id不能为空") @QueryParam("releaseTaskId") Long releaseTaskId) { + String username = SecurityFilter.getLoginUsername(request); + Message message; + try { + ResponseConvertOrchestrator response = publishService.getStatus(username, releaseTaskId.toString()); + if (null != response.getResponse()) { + String status = response.getResponse().getJobStatus().toString(); + status = StringUtils.isNotBlank(status) ? status.toLowerCase() : status; + //将发布失败原因，返回前端 + if ("failed".equalsIgnoreCase(status)) { + message = Message.error("发布失败:" + response.getResponse().getMessage()).data("status", status); + } else if (StringUtils.isNotBlank(status)) { + message = Message.ok("获取进度成功").data("status", status); + } else { + LOGGER.error("status is null or empty, failed to get status"); + message = Message.error("获取进度失败"); + } + } else { + LOGGER.error("status is null or empty, failed to get status"); + message = Message.error("获取进度失败"); + } + } catch (final Throwable t) { + LOGGER.error("Failed to get release status for {}", releaseTaskId, t); + message = Message.error("发布异常:" + t.getMessage()); + } + return Message.messageToResponse(message); + } + + + /** + * 更新工作流的基本信息，不包括更新Json,BML版本等 + * @param req + * @param json + * @return + * @throws DSSErrorException + */ + + @POST + @Path("/updateFlowBaseInfo") +// @ProjectPrivChecker + public Response updateFlowBaseInfo(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { + Long flowID = json.get("id").getLongValue(); + String name = json.get("name") == null ? null : json.get("name").getTextValue(); + String description = json.get("description") == null ? null : json.get("description").getTextValue(); + String uses = json.get("uses") == null ? null : json.get("uses").getTextValue(); +// ioManager.checkeIsExecuting(projectVersionID); + // TODO: 2019/6/13 projectVersionID的更新校验 + //这里可以不做事务 + DSSFlow dssFlow = new DSSFlow(); + dssFlow.setId(flowID); + dssFlow.setName(name); + dssFlow.setDescription(description); + dssFlow.setUses(uses); + flowService.updateFlowBaseInfo(dssFlow); + return Message.messageToResponse(Message.ok()); + } + + /** + * 读取工作流的Json数据，提供给前端渲染 + * @param req + * @param flowID + * @return + * @throws DSSErrorException + */ + + @GET + @Path("/get") + public Response get(@Context HttpServletRequest req, @QueryParam("flowId") Long flowID + ) throws DSSErrorException { + // TODO: 2019/5/23 id空值判断 + String username = SecurityFilter.getLoginUsername(req); + DSSFlow DSSFlow; + DSSFlow = flowService.getLatestVersionFlow(flowID); +// if (!username.equals(DSSFlow.getCreator())) { +// return Message.messageToResponse(Message.ok("不可以访问别人工作流")); +// } + return Message.messageToResponse(Message.ok().data("flow", DSSFlow)); + } + + +// @GET +// @Path("/product/get") +// public Response productGet(@Context HttpServletRequest req, @QueryParam("id") Long flowID) throws ErrorException { +// DSSFlow DSSFlow; +// DSSFlow = flowService.getLatestVersionFlow(flowID); +//// dwsFlow=flowService.genBusinessTagForNode(dwsFlow); +// return Message.messageToResponse(Message.ok().data("flow", DSSFlow)); +// } + + + @POST + @Path("/deleteFlow") +// @ProjectPrivChecker + public Response deleteFlow(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { + Long flowID = json.get("id").getLongValue(); + boolean sure = json.get("sure") != null && json.get("sure").getBooleanValue(); + // TODO: 2019/6/13 projectVersionID的更新校验 + //state为true代表曾经发布过 + if (flowService.getFlowByID(flowID).getState() && !sure) { + return Message.messageToResponse(Message.ok().data("warmMsg", "该工作流曾经发布过，删除将会将该工作流的所有版本都删除，是否继续？")); + } +// ioManager.checkeIsExecuting(projectVersionID); + flowService.batchDeleteFlow(Arrays.asList(flowID)); + return Message.messageToResponse(Message.ok()); + } + + /** + * 工作流保存接口，如工作流Json内容有变化，会更新工作流的Json内容 + * @param req + * @param json + * @return + * @throws DSSErrorException + * @throws IOException + */ + + @POST + @Path("/saveFlow") +// @ProjectPrivChecker + public Response saveFlow(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException, IOException { + Long flowID = json.get("id").getLongValue(); + String jsonFlow = json.get("json").getTextValue(); + String workspaceName = json.get("workspaceName").getTextValue(); + String projectName = json.get("projectName").getTextValue(); + String userName = SecurityFilter.getLoginUsername(req); + //String comment = json.get("comment") == null?"保存更新":json.get("comment").getTextValue(); + //目前保存comment不使用了,发布才有comment + String comment = null; +// ioManager.checkeIsExecuting(projectVersionID); + // TODO: 2020/6/9 为了cs和bml，加锁 + String version = null; + String newFlowEditLock = null; + synchronized (DSSWorkFlowConstant.saveFlowLock.intern(flowID)) { + version = flowService.saveFlow(flowID, jsonFlow, comment, userName, workspaceName, projectName); + } + return Message.messageToResponse(Message.ok().data("flowVersion", version).data("flowEditLock", newFlowEditLock)); + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/NodeRestfulApi.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/NodeRestfulApi.java new file mode 100644 index 000000000..1f559f32c --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/restful/NodeRestfulApi.java @@ -0,0 +1,297 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.restful; + + +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlySSOAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.standard.sso.utils.SSOHelper; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.cs.DSSCSHelper; +import com.webank.wedatasphere.dss.workflow.entity.CommonAppConnNode; +import com.webank.wedatasphere.dss.workflow.entity.ContentLanguage; +import com.webank.wedatasphere.dss.workflow.entity.NodeGroup; +import com.webank.wedatasphere.dss.workflow.entity.NodeInfo; +import com.webank.wedatasphere.dss.workflow.entity.NodeUi; +import com.webank.wedatasphere.dss.workflow.entity.NodeUiValidate; +import com.webank.wedatasphere.dss.workflow.entity.vo.NodeGroupVO; +import com.webank.wedatasphere.dss.workflow.entity.vo.NodeInfoVO; +import com.webank.wedatasphere.dss.workflow.entity.vo.NodeUiVO; +import com.webank.wedatasphere.dss.workflow.entity.vo.NodeUiValidateVO; +import com.webank.wedatasphere.dss.workflow.function.DomainSupplier; +import com.webank.wedatasphere.dss.workflow.function.FunctionInvoker; +import com.webank.wedatasphere.dss.workflow.function.FunctionPool; +import com.webank.wedatasphere.dss.workflow.service.DSSFlowService; +import com.webank.wedatasphere.dss.workflow.service.WorkflowNodeService; +import com.webank.wedatasphere.linkis.common.conf.Configuration; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import com.webank.wedatasphere.linkis.server.Message; +import com.webank.wedatasphere.linkis.server.security.SecurityFilter; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import javax.servlet.http.HttpServletRequest; +import javax.ws.rs.Consumes; +import javax.ws.rs.GET; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.Produces; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.BeanUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +@Component +@Path("/dss/workflow") +@Produces(MediaType.APPLICATION_JSON) +@Consumes(MediaType.APPLICATION_JSON) +public class NodeRestfulApi { + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private FunctionInvoker functionInvoker; + + @Autowired + private DSSFlowService DSSFlowService; + + @Autowired + private WorkflowNodeService workflowNodeService; + + @GET + @Path("/listNodeType") + public Response listNodeType(@Context HttpServletRequest req) { + DomainSupplier supplier = internationalization(req, NodeGroup::getNameEn, NodeGroup::getName); + List groupVOS = new ArrayList<>(); + //cache + List groups = workflowNodeService.listNodeGroups(); + for (NodeGroup group : groups) { + NodeGroupVO nodeGroupVO = new NodeGroupVO(); + BeanUtils.copyProperties(group, nodeGroupVO); + nodeGroupVO.setTitle(supplier.get(group)); + nodeGroupVO.setChildren(group.getNodes().stream().map(n -> { + try { + return transfer(n, req); + } catch (AppStandardErrorException e) { + e.printStackTrace(); + return null; + } + }).collect(Collectors.toList())); + groupVOS.add(nodeGroupVO); + } + groupVOS = groupVOS.stream().sorted(NodeGroupVO::compareTo).collect(Collectors.toList()); + return Message.messageToResponse(Message.ok().data("nodeTypes", groupVOS)); + } + + private DomainSupplier internationalization(HttpServletRequest req, + DomainSupplier supplier1, + DomainSupplier supplier2) { + String language = req.getHeader("Content-language"); + if (language != null) {language = language.trim();} + if (ContentLanguage.en.getName().equals(language)) + {return supplier1;} + else { + return supplier2; + } + } + + private NodeUiValidateVO transfer(NodeUiValidate v, HttpServletRequest req) { + DomainSupplier supplier = internationalization(req, NodeUiValidate::getErrorMsgEn, + NodeUiValidate::getErrorMsg); + NodeUiValidateVO nodeUiValidateVO = new NodeUiValidateVO(); + BeanUtils.copyProperties(v, nodeUiValidateVO); + nodeUiValidateVO.setMessage(supplier.get(v)); + return nodeUiValidateVO; + } + + private NodeInfoVO transfer(NodeInfo nodeInfo, HttpServletRequest req) throws AppStandardErrorException { + NodeInfoVO nodeInfoVO = new NodeInfoVO(); + BeanUtils.copyProperties(nodeInfo, nodeInfoVO); + nodeInfoVO.setTitle(nodeInfo.getName()); + nodeInfoVO.setType(nodeInfo.getNodeType()); + nodeInfoVO.setImage(nodeInfo.getIcon()); + DomainSupplier descriptionSupplier = internationalization(req, NodeUi::getDescriptionEn, NodeUi::getDescription); + DomainSupplier lableNameSupplier = internationalization(req, NodeUi::getLableNameEn, NodeUi::getLableName); + ArrayList nodeUiVOS = new ArrayList<>(); + for (NodeUi nodeUi : nodeInfo.getNodeUis()) { + NodeUiVO nodeUiVO = new NodeUiVO(); + BeanUtils.copyProperties(nodeUi, nodeUiVO); + nodeUiVO.setDesc(descriptionSupplier.get(nodeUi)); + nodeUiVO.setLableName(lableNameSupplier.get(nodeUi)); + nodeUiVO.setNodeUiValidateVOS(nodeUi.getNodeUiValidates().stream().map(v -> transfer(v, req)).sorted(NodeUiValidateVO::compareTo).collect(Collectors.toList())); + nodeUiVOS.add(nodeUiVO); + } + nodeUiVOS.sort(NodeUiVO::compareTo); + nodeInfoVO.setNodeUiVOS(nodeUiVOS); + //cache + AppConn applicationAppConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + if (applicationAppConn instanceof OnlySSOAppConn) { + SSOIntegrationStandard standard =((OnlySSOAppConn) applicationAppConn).getOrCreateSSOStandard(); + if(null !=standard) { + SSOUrlBuilderOperation ssoUrlBuilderOperation=standard.getSSOBuilderService().createSSOUrlBuilderOperation(); + ssoUrlBuilderOperation.setDSSUrl(Configuration.GATEWAY_URL().getValue()); + //todo add redirect url by labels + ssoUrlBuilderOperation.setReqUrl(applicationAppConn.getAppDesc().getAppInstances().get(0).getBaseUrl()); + String redirectUrl =ssoUrlBuilderOperation.getBuiltUrl(); + if (redirectUrl != null) { + nodeInfoVO.setJumpUrl(redirectUrl); + } + } + } + return nodeInfoVO; + } + + @POST + @Path("/createAppConnNode") + public Response createExternalNode(@Context HttpServletRequest req, Map json) throws DSSErrorException, IllegalAccessException, ExternalOperationFailedException, InstantiationException { + String userName = SecurityFilter.getLoginUsername(req); + Workspace workspace = SSOHelper.getWorkspace(req); + Long projectID = Long.parseLong(json.get("projectID").toString()); + String nodeType = json.get("nodeType").toString(); + Long flowID = Long.parseLong(json.get("flowID").toString()); + Map params = (Map) json.get("params"); + logger.info("CreateExternalNode request params is " + params + ",nodeType:" + nodeType); + CommonAppConnNode node = new CommonAppConnNode(); + node.setProjectId(projectID); + node.setNodeType(nodeType); + node.setFlowId(flowID); + node.setProjectId(projectID); + //update by peaceWong add nodeID to appConnNode + Object nodeID = json.get(CSCommonUtils.NODE_ID); + if (null != nodeID) { + node.setId(nodeID.toString()); + } + //update by shanhuang json中解析获取contextID,然后放到请求参数中 + DSSFlow DSSFlow = DSSFlowService.getLatestVersionFlow(flowID); + String flowContent = DSSFlow.getFlowJson(); + String ContextIDStr = DSSCSHelper.getContextIDStrByJson(flowContent); + params.put(CSCommonUtils.CONTEXT_ID_STR, ContextIDStr); + //补个flowName的信息..如果这里没查询就直接让前台传了 + node.setFlowName(DSSFlow.getName()); + //补充json信息,方便appconn去解析获取响应的值 + params.put("json", flowContent); + String label = ((Map) json.get(DSSCommonUtils.DSS_LABELS_KEY)).get("route").toString(); + params.put(DSSCommonUtils.DSS_LABELS_KEY, label); + params.put("workspace", workspace); + functionInvoker.nodeServiceFunction(userName, params, node, FunctionPool.createNode); + return Message.messageToResponse(Message.ok().data("result", node.getJobContent())); + } + + @POST + @Path("/updateAppConnNode") + public Response updateExternalNode(@Context HttpServletRequest req, Map json) throws IllegalAccessException, ExternalOperationFailedException, InstantiationException { + String userName = SecurityFilter.getLoginUsername(req); + Workspace workspace = SSOHelper.getWorkspace(req); + Long projectID = Long.parseLong(json.get("projectID").toString()); + String nodeType = json.get("nodeType").toString(); + Map params = (Map) json.get("params"); + logger.info("UpdateExternalNode request params is " + params + ",nodeType:" + nodeType); + CommonAppConnNode node = new CommonAppConnNode(); + node.setProjectId(projectID); + node.setNodeType(nodeType); + node.setProjectId(projectID); + String label = ((Map) json.get(DSSCommonUtils.DSS_LABELS_KEY)).get("route").toString(); + params.put(DSSCommonUtils.DSS_LABELS_KEY, label); + params.put("workspace", workspace); + functionInvoker.nodeServiceFunction(userName, params, node, FunctionPool.updateNode); + return Message.messageToResponse(Message.ok().data("result", node.getJobContent())); + } + + @POST + @Path("/deleteAppConnNode") + public Response deleteExternalNode(@Context HttpServletRequest req, Map json) throws IllegalAccessException, ExternalOperationFailedException, InstantiationException { + String userName = SecurityFilter.getLoginUsername(req); + Workspace workspace = SSOHelper.getWorkspace(req); + Long projectID = Long.parseLong(json.get("projectID").toString()); + String nodeType = json.get("nodeType").toString(); + Map params = (Map) json.get("params"); + logger.info("DeletepwdExternalNode request params is " + params + ",nodeType:" + nodeType); + CommonAppConnNode node = new CommonAppConnNode(); + node.setProjectId(projectID); + node.setNodeType(nodeType); + String label = ((Map) json.get(DSSCommonUtils.DSS_LABELS_KEY)).get("route").toString(); + params.put(DSSCommonUtils.DSS_LABELS_KEY, label); + params.put("workspace", workspace); + functionInvoker.nodeServiceFunction(userName, params, node, FunctionPool.deleteNode); + return Message.messageToResponse(Message.ok().data("result", node.getJobContent())); + } + + @POST + @Path("/batchDeleteAppConnNode") + public Response batchDeleteAppConnNode(@Context HttpServletRequest req, Map reqJson) throws IllegalAccessException, ExternalOperationFailedException, InstantiationException { + String userName = SecurityFilter.getLoginUsername(req); + Workspace workspace = SSOHelper.getWorkspace(req); + List> jsonList = (List>) reqJson.get("nodes"); + jsonList.stream().forEach(json ->{ + Long projectID = Long.parseLong(json.get("projectID").toString()); + String nodeType = json.get("nodeType").toString(); + Map params = (Map) json.get("params"); + logger.info("DeletepwdExternalNode request params is " + params + ",nodeType:" + nodeType); + CommonAppConnNode node = new CommonAppConnNode(); + node.setProjectId(projectID); + node.setNodeType(nodeType); + String label = ((Map) json.get("labels")).get("route").toString(); + params.put("labels", label); + params.put("workspace", workspace); + try { + functionInvoker.nodeServiceFunction(userName, params, node, FunctionPool.deleteNode); + } catch (IllegalAccessException e) { + logger.error("Delete appconn node failde!",e); + } catch (ExternalOperationFailedException e) { + logger.error("Delete appconn node failde!",e); + } catch (InstantiationException e) { + logger.error("Delete appconn node failde!",e); + } + }); + + return Message.messageToResponse(Message.ok().data("result","success")); + } + + @POST + @Path("/getAppConnNodeUrl") + public Response getAppConnNodeUrl(@Context HttpServletRequest req, Map json) throws IllegalAccessException, ExternalOperationFailedException, InstantiationException { + String userName = SecurityFilter.getLoginUsername(req); + Workspace workspace = SSOHelper.getWorkspace(req); + Long projectID = Long.parseLong(json.get("projectID").toString()); + String nodeType = json.get("nodeType").toString(); + Map params = (Map) json.get("params"); + logger.info("getAppConnNodeUrl request params is " + params + ",nodeType:" + nodeType); + CommonAppConnNode node = new CommonAppConnNode(); + node.setProjectId(projectID); + node.setNodeType(nodeType); + String label = ((Map) json.get(DSSCommonUtils.DSS_LABELS_KEY)).get("route").toString(); + params.put(DSSCommonUtils.DSS_LABELS_KEY, label); + params.put("workspace", workspace); + String jumpUrl = workflowNodeService.getNodeJumpUrl(params, node); + return Message.messageToResponse(Message.ok().data("jumpUrl", jumpUrl)); + } + + +} + diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJob.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJob.java new file mode 100644 index 000000000..f34c8b116 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJob.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public abstract class DssJob implements Runnable { + + protected Logger logger = LoggerFactory.getLogger(this.getClass()); + protected String user; + protected Long id; + protected DssJobListener dssJobListener; + protected DssJobStatus status; + + public DssJobListener getDssJobListener() { + return dssJobListener; + } + + public void setDssJobListener(DssJobListener dssJobListener) { + this.dssJobListener = dssJobListener; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + public DssJobStatus getStatus() { + return status; + } + + public void setStatus(DssJobStatus status) { + this.status = status; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobDeamon.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobDeamon.java new file mode 100644 index 000000000..16dead9b9 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobDeamon.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +import java.util.concurrent.Future; + +public class DssJobDeamon extends DssJob { + private Future future; + + private int time; + + public DssJobDeamon(Future future, int time) { + this.future = future; + this.time = time; + } + + @Override + public void run() { + logger.info("id:{}的job守护线程启动", id); + try { + Thread.sleep(1000L * time); + } catch (InterruptedException e) { + logger.error("deamon被打断了", e); + Thread.currentThread().interrupt(); + } + logger.info("开始取消id:{}的job线程", id); + future.cancel(true); + logger.info("成功取消id:{}的job线程", id); + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobHook.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobHook.java new file mode 100644 index 000000000..7716a67a2 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobHook.java @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +public interface DssJobHook { + void preExecute(DssJob job); + + void postExecute(DssJob job); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobInfo.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobInfo.java new file mode 100644 index 000000000..2df13ec92 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobInfo.java @@ -0,0 +1,79 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + + +import com.webank.wedatasphere.dss.workflow.constant.DSSWorkFlowConstant; + +import java.util.Date; + +public class DssJobInfo { + + private Long id; + private DssJobStatus status; + private Date createTime; + private Date updateTime; + private String msg; + + public Boolean timeout() { + return System.currentTimeMillis() - updateTime.getTime() > (int) DSSWorkFlowConstant.CACHE_TIMEOUT.getValue(); + } + + public Boolean isExecute() { + return status != null && (status == DssJobStatus.Inited || status == DssJobStatus.Running); + } + + public String getMsg() { + return msg; + } + + public void setMsg(String msg) { + this.msg = msg; + } + + public DssJobStatus getStatus() { + return status; + } + + public void setStatus(DssJobStatus status) { + this.status = status; + } + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + public Date getUpdateTime() { + return updateTime; + } + + public void setUpdateTime(Date updateTime) { + this.updateTime = updateTime; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobListener.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobListener.java new file mode 100644 index 000000000..6d566ebe3 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobListener.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +public interface DssJobListener { + + void onJobSucceed(DssJob job, String infoMsg); + + void onJobFailed(DssJob job, String errorMsg); + + void onJobRunning(DssJob job, String infoMsg); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobManager.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobManager.java new file mode 100644 index 000000000..ef264b70e --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobManager.java @@ -0,0 +1,172 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.workflow.constant.DSSWorkFlowConstant; +import com.webank.wedatasphere.linkis.common.utils.Utils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; + + +public abstract class DssJobManager implements DssJobListener { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + private final Map cacheMap = new ConcurrentHashMap<>(); + + private final Map secondCacheMap = new ConcurrentHashMap<>();//执行完成后将放入二级缓存,只有qurey的时候才去掉 + + private List jobHooks = new ArrayList<>(); + + public void addDssJobHook(DssJobHook jobHook) { + jobHooks.add(jobHook); + } + + { + logger.info("定时线程开启..."); + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (secondCacheMap) { + secondCacheMap.entrySet().stream() + .filter(f -> (f.getValue().getStatus() == DssJobStatus.Succeed || f.getValue().getStatus() == DssJobStatus.Failed) && f.getValue().timeout()) + .forEach(f -> { + logger.info("开始remove second cache过期记录：{},更新时间{}", f.getKey(), f.getValue().getUpdateTime()); + secondCacheMap.remove(f.getKey()); + }); + } + } + }, 1, 1, TimeUnit.MINUTES); + Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { + @Override + public void run() { + synchronized (cacheMap) { + cacheMap.forEach((k, v) -> { + logger.info("cache中id为：{}状态为：{}更新时间为：{}", k, v.getStatus(), v.getUpdateTime()); + }); + } + } + }, 1, 10, TimeUnit.MINUTES); + } + + @Override + public void onJobSucceed(DssJob job, String infoMsg) { + // TODO: 2019/6/21 目前不用加锁 + job.setStatus(DssJobStatus.Succeed); + logger.info(infoMsg); + Date updateTime = new Date(); + DssJobInfo cache = cacheMap.get(job.getId()); + cache.setUpdateTime(updateTime); + cache.setStatus(DssJobStatus.Succeed); + synchronized (String.valueOf(job.getId()).intern()) { + secondCacheMap.put(job.getId(), cacheMap.remove(job.getId())); + } + jobHooks.forEach(h -> h.postExecute(job)); + } + + @Override + public void onJobFailed(DssJob job, String errorMsg) { + job.setStatus(DssJobStatus.Failed); + logger.info(errorMsg); + Date updateTime = new Date(); + DssJobInfo cache = cacheMap.get(job.getId()); + cache.setUpdateTime(updateTime); + cache.setStatus(DssJobStatus.Failed); + cache.setMsg(errorMsg); + synchronized (String.valueOf(job.getId()).intern()) { + secondCacheMap.put(job.getId(), cacheMap.remove(job.getId())); + } + jobHooks.forEach(h -> h.postExecute(job)); + } + + @Override + public void onJobRunning(DssJob job, String infoMsg) { + job.setStatus(DssJobStatus.Running); + logger.info(infoMsg); + Date updateTime = new Date(); + DssJobInfo cache = cacheMap.get(job.getId()); + cache.setUpdateTime(updateTime); + cache.setStatus(DssJobStatus.Running); + } + + protected void addPublishCache(DssJobInfo job) throws DSSErrorException { + job.setStatus(DssJobStatus.Inited); + job.setCreateTime(new Date()); + job.setUpdateTime(new Date()); + cacheMap.put(job.getId(), job); + } + + private DssJobInfo getJobInfoCache(Long id) { + return cacheMap.get(id); + } + + private DssJobInfo getSecondJobInfoCache(Long id) { + return secondCacheMap.get(id); + } + + public void checkeIsExecuting(Long id) throws DSSErrorException { + DssJobInfo jobInfo = getJobInfoCache(id); + DssJobInfo seCondInfo = getSecondJobInfoCache(id); + if (jobInfo != null && jobInfo.isExecute() || seCondInfo != null) { + DssJobStatus status = jobInfo == null ? seCondInfo.getStatus() : jobInfo.getStatus(); + logger.info("job执行中,id:{}，状态为{}", id, status); + throw new DSSErrorException(90014, "job执行中，不允许执行当前操作"); + } + } + + private DssJobInfo createJobInfo(Long id) { + DssJobInfo jobInfo = new DssJobInfo(); + jobInfo.setId(id); + return jobInfo; + } + + public void submit(DssJob job) throws DSSErrorException { + synchronized (String.valueOf(job.getId()).intern()) { + checkeIsExecuting(job.getId()); + addPublishCache(createJobInfo(job.getId())); + } + job.setStatus(DssJobStatus.Inited); + jobHooks.forEach(h -> h.preExecute(job)); + Future submit = DssJobThreadPool.get().submit(job); + DssJobDeamon deamon = new DssJobDeamon(submit, (int) DSSWorkFlowConstant.PUBLISH_TIMEOUT.getValue()); + deamon.setId(job.getId()); + DssJobThreadPool.getDeamon().execute(deamon); + } + + public DssJobInfo getJobStatus(Long id) throws DSSErrorException { + synchronized (String.valueOf(id).intern()) { + DssJobInfo jobInfo = getJobInfoCache(id); + if (jobInfo != null) { + return jobInfo; + } else { + DssJobInfo remove = secondCacheMap.remove(id); + if (remove == null) throw new DSSErrorException(90021, String.format("the id %s is not exist", id)); + return remove; + } + } + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobStatus.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobStatus.java new file mode 100644 index 000000000..92209e23b --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobStatus.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +public enum DssJobStatus { + Inited,Running,Succeed,Failed +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobThreadPool.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobThreadPool.java new file mode 100644 index 000000000..15d4f0564 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/DssJobThreadPool.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +import com.webank.wedatasphere.linkis.common.utils.Utils; + +import java.util.concurrent.ExecutorService; + +public class DssJobThreadPool { + + private static ExecutorService executorService = Utils.newFixedThreadPool(1000, "project-publish", false); + private static ExecutorService executorServiceDeamon = Utils.newFixedThreadPool(1000, "project-publish-deamon", true); + + public static ExecutorService get() { + return executorService; + } + + public static ExecutorService getDeamon() { + return executorServiceDeamon; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/JobParams.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/JobParams.java new file mode 100644 index 000000000..4f9af649c --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/scheduler/JobParams.java @@ -0,0 +1,20 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.scheduler; + +public interface JobParams { +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/DSSFlowService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/DSSFlowService.java new file mode 100644 index 000000000..8425998a5 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/DSSFlowService.java @@ -0,0 +1,84 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; + +import java.io.IOException; +import java.util.List; + +public interface DSSFlowService { + DSSFlow getFlowByID(Long id); + + DSSFlow getFlowWithJsonAndSubFlowsByID(Long rootFlowId); + + DSSFlow addFlow(DSSFlow DSSFlow, String contextIDStr) throws DSSErrorException; + + DSSFlow addSubFlow(DSSFlow DSSFlow, Long parentFlowID, String contextIDStr) throws DSSErrorException; + + /** + * 通过flowID获取最新版本的dwsFlow，版本信息在latestVersion + * @param flowID + * @return + */ + DSSFlow getLatestVersionFlow(Long flowID) throws DSSErrorException; + + /** + * 通过flowID和某个版本号，获取一个dwsFlow，版本信息在versions数组中的第一个元素 + * @param flowID + * @return + */ + DSSFlow getOneVersionFlow(Long flowID); + + /** + * 通过dwsFlow对象拿到最新的json，其实这里只要个flowID应该就可以了 + * @param DSSFlow + * @return + */ +/* String getLatestJsonByFlow(DWSFlow dwsFlow); + + DWSFlow getLatestVersionFlow(Long ProjectID,String flowName);*/ + + void updateFlowBaseInfo(DSSFlow DSSFlow) throws DSSErrorException; + + + void batchDeleteFlow(List flowIDlist); + + String saveFlow(Long flowID, + String jsonFlow, + String comment, + String userName, + String workspaceName, + String projectName + ) throws DSSErrorException; + + Integer getParentRank(Long flowID); + + DSSFlow copyRootFlow(Long rootFlowId,String userName,String workspaceName,String projectName,String version,String contextIdStr) throws DSSErrorException, IOException; + +// DWSFlowVersion getLatestVersionByFlowIDAndProjectVersionID(Long flowID, Long projectVersionID); + + Long getParentFlowID(Long id); + +// AppMap getFlowAppMap(long projectID); +// +// DWSFlow genBusinessTagForNode(DWSFlow dwsFlow) throws DSSErrorException; +// +// void flowVersionIncr(Long flowID, String userName, String comment, Long newProjectVersionID, Long oldProjectVersionID, +// DWSProject dwsProject,String workspace); +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/PublishService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/PublishService.java new file mode 100644 index 000000000..09f7cbaa0 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/PublishService.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service; + +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseConvertOrchestrator; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import java.util.Map; + +public interface PublishService { + + + String submitPublish(String publishUser, Long workflowId, + Map dssLabel, Workspace workspace, String comment) throws Exception; + + + ResponseConvertOrchestrator getStatus(String username, String taskId) throws Exception; +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/WorkflowNodeService.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/WorkflowNodeService.java new file mode 100644 index 000000000..569a228ae --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/WorkflowNodeService.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.workflow.entity.AbstractAppConnNode; +import com.webank.wedatasphere.dss.workflow.entity.NodeGroup; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +public interface WorkflowNodeService { + + List listNodeGroups(); + + /** + * 根据参数创建外部节点 + * @param node 节点信息 + * @return 返回jobContent的Map，工作流会将该Map存储起来，作为该节点的关键关联信息，用于后续的CRUD和执行。 + */ + Map createNode(String userName, AbstractAppConnNode node + ) throws ExternalOperationFailedException; + + void deleteNode(String userName, AbstractAppConnNode node) throws ExternalOperationFailedException; + + Map updateNode(String userName, AbstractAppConnNode node) throws ExternalOperationFailedException; + + default Map refresh(String userName,AbstractAppConnNode node) { + return null; + } + + default void copyNode(String userName, AbstractAppConnNode newNode, AbstractAppConnNode oldNode) { } + + default void setNodeReadOnly(String userName, AbstractAppConnNode node) {} + + default List listNodes(String userName, AbstractAppConnNode node) { + return new ArrayList<>(); + } + + default Map exportNode(String userName, AbstractAppConnNode node) { + return null; + } + + default Map importNode(String userName, AbstractAppConnNode node, Map resourceMap, Workspace workspace, String orcVersion) throws Exception { + return null; + } + + String getNodeJumpUrl(Map params, AbstractAppConnNode node) throws ExternalOperationFailedException; + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/DSSFlowServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/DSSFlowServiceImpl.java new file mode 100644 index 000000000..068a583fb --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/DSSFlowServiceImpl.java @@ -0,0 +1,416 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service.impl; + + +import com.google.gson.Gson; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.contextservice.service.ContextService; +import com.webank.wedatasphere.dss.contextservice.service.impl.ContextServiceImpl; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow; +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlowRelation; +import com.webank.wedatasphere.dss.workflow.common.parser.WorkFlowParser; +import com.webank.wedatasphere.dss.workflow.dao.FlowMapper; +import com.webank.wedatasphere.dss.workflow.io.input.NodeInputService; +import com.webank.wedatasphere.dss.workflow.lock.Lock; +import com.webank.wedatasphere.dss.workflow.service.BMLService; +import com.webank.wedatasphere.dss.workflow.service.DSSFlowService; +import com.webank.wedatasphere.linkis.cs.common.utils.CSCommonUtils; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.BeanUtils; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.dao.DuplicateKeyException; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.io.IOException; +import java.util.*; +import java.util.stream.Collectors; + +@Service +public class DSSFlowServiceImpl implements DSSFlowService { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private FlowMapper flowMapper; + + + @Autowired + private NodeInputService nodeInputService; + +// @Autowired +// private EventRelationMapper eventRelationMapper; + + @Autowired + private WorkFlowParser workFlowParser; + + @Autowired + private BMLService bmlService; + + +// @Autowired(required = false) +// private LockMapper lockMapper; + + private static ContextService contextService = ContextServiceImpl.getInstance(); + + @Override + public DSSFlow getFlowByID(Long id) { + return flowMapper.selectFlowByID(id); + } + + @Override + public DSSFlow getFlowWithJsonAndSubFlowsByID(Long rootFlowId) { + + return genDSSFlowTree(rootFlowId); + } + + + private DSSFlow genDSSFlowTree(Long parentFlowId) { + DSSFlow cyFlow = flowMapper.selectFlowByID(parentFlowId); + + String userName = cyFlow.getCreator(); + Map query = bmlService.query(userName, cyFlow.getResourceId(), cyFlow.getBmlVersion()); + cyFlow.setFlowJson(query.get("string").toString()); + + List subFlowIDs = flowMapper.selectSubFlowIDByParentFlowID(parentFlowId); + for (Long subFlowID : subFlowIDs) { + if (cyFlow.getChildren() == null) { + cyFlow.setChildren(new ArrayList()); + } + DSSFlow SubFlow = genDSSFlowTree(subFlowID); + + cyFlow.addChildren(SubFlow); + } + return cyFlow; + } + + + @Lock + @Transactional(rollbackFor = DSSErrorException.class) + @Override + public DSSFlow addFlow(DSSFlow dssFlow, + String contextID) throws DSSErrorException { + try { + flowMapper.insertFlow(dssFlow); + } catch (DuplicateKeyException e) { + logger.info(e.getMessage()); + throw new DSSErrorException(90003, "工作流名不能重复"); + } + Map contextIDMap = new HashMap<>(); + String userName = dssFlow.getCreator(); + if (StringUtils.isNotBlank(contextID)) { + contextIDMap.put(CSCommonUtils.CONTEXT_ID_STR, contextID); + } else { + logger.error("Generate contextID null."); + } + String jsonFlow = new Gson().toJson(contextIDMap); + Map bmlReturnMap = bmlService.upload(userName, jsonFlow, UUID.randomUUID().toString() + ".json", + dssFlow.getName()); + + dssFlow.setResourceId(bmlReturnMap.get("resourceId").toString()); + + dssFlow.setBmlVersion(bmlReturnMap.get("version").toString()); + Long parentFlowID = null; + if (!dssFlow.getRootFlow()) { + parentFlowID = flowMapper.getParentFlowID(dssFlow.getId()); + } + //todo update dssflow + contextService.checkAndSaveContext(jsonFlow, String.valueOf(parentFlowID)); + flowMapper.updateFlowInputInfo(dssFlow); + return dssFlow; + } + + @Lock + @Transactional(rollbackFor = DSSErrorException.class) + @Override + public DSSFlow addSubFlow(DSSFlow DSSFlow, Long parentFlowID, String contextIDStr) throws DSSErrorException { + DSSFlow parentFlow = flowMapper.selectFlowByID(parentFlowID); + DSSFlow.setProjectID(parentFlow.getProjectID()); + DSSFlow subFlow = addFlow(DSSFlow, contextIDStr); + //数据库中插入关联信息 + flowMapper.insertFlowRelation(subFlow.getId(), parentFlowID); + return subFlow; + } + + @Override + public DSSFlow getLatestVersionFlow(Long flowID) { + DSSFlow DSSFlow = getFlowByID(flowID); + //todo update + String userName = DSSFlow.getCreator(); + Map query = bmlService.query(userName, DSSFlow.getResourceId(), DSSFlow.getBmlVersion()); + DSSFlow.setFlowJson(query.get("string").toString()); + return DSSFlow; + } + + @Override + public DSSFlow getOneVersionFlow(Long flowID) { + DSSFlow DSSFlow = getFlowByID(flowID); + + String userName = DSSFlow.getCreator(); + Map query = bmlService.query(userName, DSSFlow.getResourceId(), DSSFlow.getBmlVersion()); + DSSFlow.setFlowJson(query.get("string").toString()); + return DSSFlow; + } + + @Lock + @Transactional(rollbackFor = DSSErrorException.class) + @Override + public void updateFlowBaseInfo(DSSFlow DSSFlow) throws DSSErrorException { + try { + flowMapper.updateFlowBaseInfo(DSSFlow); + } catch (DuplicateKeyException e) { + logger.info(e.getMessage()); + throw new DSSErrorException(90003, "工作流名不能重复"); + } + } + + @Lock + @Transactional(rollbackFor = DSSErrorException.class) + @Override + public void batchDeleteFlow(List flowIDlist) { + flowIDlist.stream().forEach(f -> { + deleteFlow(f); + }); + } + + /*@Lock*/ + @Transactional(rollbackFor = {DSSErrorException.class}) + @Override + public String saveFlow(Long flowID, + String jsonFlow, + String comment, + String userName, + String workspaceName, + String projectName + ) throws DSSErrorException { + + DSSFlow dssFlow = flowMapper.selectFlowByID(flowID); + String creator = dssFlow.getCreator(); + if (StringUtils.isNotEmpty(creator)) { + userName = creator; + } + + String resourceId = dssFlow.getResourceId(); + + Long parentFlowID = flowMapper.getParentFlowID(flowID); + + // 这里不要检查ContextID具体版本等，只要存在就不创建 2020-0423 + jsonFlow = contextService.checkAndCreateContextID(jsonFlow, dssFlow.getBmlVersion(), + workspaceName, projectName, dssFlow.getName(), userName, false); + + + Map bmlReturnMap = bmlService.update(userName, resourceId, jsonFlow); + + dssFlow.setId(flowID); + dssFlow.setHasSaved(true); + dssFlow.setDescription(comment); + dssFlow.setResourceId(bmlReturnMap.get("resourceId").toString()); + dssFlow.setBmlVersion(bmlReturnMap.get("version").toString()); + //todo 数据库增加版本更新 + flowMapper.updateFlowInputInfo(dssFlow); + + try { + contextService.checkAndSaveContext(jsonFlow, String.valueOf(parentFlowID)); + } catch (DSSErrorException e) { + logger.error("Failed to saveContext: ", e); + } + + String version = bmlReturnMap.get("version").toString(); + + return version; + } + + @Override + public Integer getParentRank(Long parentFlowID) { + return getFlowByID(parentFlowID).getRank(); + } + + @Override + public Long getParentFlowID(Long flowID) { + return flowMapper.getParentFlowID(flowID); + } + + @Deprecated + private Integer getParentFlowIDByFlowID(Long flowID, Integer rank) { + Long parentFlowID = flowMapper.selectParentFlowIDByFlowID(flowID); + if (parentFlowID != null) { + rank++; + getParentFlowIDByFlowID(parentFlowID, rank); + } + return rank; + } + + public void deleteFlow(Long flowId) { + List subFlowIDs = flowMapper.selectSubFlowIDByParentFlowID(flowId); + for (Long subFlowID : subFlowIDs) { + deleteFlow(subFlowID); + } + for (Long subFlowID : subFlowIDs) { + deleteDWSDB(subFlowID); + // TODO: 2019/6/5 wtss发布过的工作流的删除？ + // TODO: 2019/6/5 json中资源的删除 + // TODO: 2019/6/5 事务的保证 + } + deleteDWSDB(flowId); + } + + private void deleteDWSDB(Long flowID) { + + flowMapper.deleteFlowBaseInfo(flowID); + flowMapper.deleteFlowRelation(flowID); + //第一期没有工作流的发布，所以不需要删除dws工作流的发布表 + } + + + @Override + public DSSFlow copyRootFlow(Long rootFlowId, String userName, String workspaceName, String projectName, String version,String contextIdStr) throws DSSErrorException, IOException { + DSSFlow dssFlow = flowMapper.selectFlowByID(rootFlowId); + DSSFlow rootFlowWithSubFlows = copyFlowAndSetSubFlowInDB(dssFlow, userName); + updateFlowJson(userName, workspaceName, projectName, rootFlowWithSubFlows, version, null,contextIdStr); + DSSFlow copyFlow = flowMapper.selectFlowByID(rootFlowWithSubFlows.getId()); + return copyFlow; + } + + + private DSSFlow copyFlowAndSetSubFlowInDB(DSSFlow DSSFlow, + String userName) { + DSSFlow cyFlow = new DSSFlow(); + BeanUtils.copyProperties(DSSFlow, cyFlow, "children", "flowVersions"); + //封装flow信息 + cyFlow.setCreator(userName); + cyFlow.setCreateTime(new Date()); + cyFlow.setId(null); + flowMapper.insertFlow(cyFlow); + List subFlowIDs = flowMapper.selectSubFlowIDByParentFlowID(DSSFlow.getId()); + for (Long subFlowID : subFlowIDs) { + DSSFlow subDSSFlow = flowMapper.selectFlowByID(subFlowID); + if (DSSFlow.getChildren() == null) { + DSSFlow.setChildren(new ArrayList()); + } + DSSFlow copySubFlow = copyFlowAndSetSubFlowInDB(subDSSFlow, userName); + persistenceFlowRelation(copySubFlow.getId(), cyFlow.getId()); + cyFlow.addChildren(copySubFlow); + } + return cyFlow; + } + + private void updateFlowJson(String userName, String workspaceName, String projectName, DSSFlow rootFlow, String version, Long parentFlowId,String contextIdStr) throws DSSErrorException, IOException { + String flowJson = bmlService.readFlowJsonFromBML(userName, rootFlow.getResourceId(), rootFlow.getBmlVersion()); + //如果包含subflow,需要一同导入subflow内容，并更新parrentflow的json内容 + // TODO: 2020/7/31 优化update方法里面的saveContent + String updateFlowJson = updateFlowContextId(flowJson,contextIdStr); + updateFlowJson = updateWorkFlowNodeJson(userName, projectName, updateFlowJson, rootFlow); + List subFlows = rootFlow.getChildren(); + if (subFlows != null) { + for (DSSFlow subflow : subFlows) { + updateFlowJson(userName, workspaceName, projectName, subflow, version, rootFlow.getId(),contextIdStr); + } + } + DSSFlow updateDssFlow = uploadFlowJsonToBml(userName, projectName, rootFlow, updateFlowJson); + //todo add dssflow to database + flowMapper.updateFlowInputInfo(updateDssFlow); + contextService.checkAndSaveContext(updateFlowJson, String.valueOf(parentFlowId)); + } + + + private DSSFlow uploadFlowJsonToBml(String userName, String projectName, DSSFlow dssFlow, String flowJson) { + + //上传文件获取resourceId和version save应该是已经有 + Map bmlReturnMap; + + //上传文件获取resourceId和version save应该是已经有 + bmlReturnMap = bmlService.upload(userName, flowJson, UUID.randomUUID().toString() + ".json", projectName); + + dssFlow.setCreator(userName); + dssFlow.setBmlVersion(bmlReturnMap.get("version").toString()); + dssFlow.setResourceId(bmlReturnMap.get("resourceId").toString()); + dssFlow.setDescription("copy update workflow"); + dssFlow.setSource("copy更新"); + + //version表中插入数据 + return dssFlow; + } + + + private String updateFlowContextId(String flowJson,String contextIdStr) throws IOException { +// String parentFlowIdStr = null; +// if (parentFlowId != null) { +// parentFlowIdStr = parentFlowId.toString(); +// } +// String contextID = contextService.checkAndInitContext(flowJson, parentFlowIdStr, workspace, projectName, flowName, flowVersion, userName); + String updatedFlowJson = workFlowParser.updateFlowJsonWithKey(flowJson, "contextID", contextIdStr); + return updatedFlowJson; + } + + + private String updateWorkFlowNodeJson(String userName, String projectName, String flowJson, DSSFlow DSSFlow) throws DSSErrorException, IOException { + List nodeJsonList = workFlowParser.getWorkFlowNodesJson(flowJson); + if (nodeJsonList == null) { + throw new DSSErrorException(90073, "工作流内没有工作流节点，导入失败" + DSSFlow.getName()); + } + String updateContextId = workFlowParser.getValueWithKey(flowJson, "contextID"); + if (nodeJsonList.size() == 0) { + return flowJson; + } + List subflows = (List) DSSFlow.getChildren(); + List> nodeJsonListRes = new ArrayList<>(); + if (nodeJsonList != null && nodeJsonList.size() > 0) { + for (String nodeJson : nodeJsonList) { + String updateNodeJson = nodeJson; + + Map nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(nodeJson, Map.class); + //更新subflowID + String nodeType = nodeJsonMap.get("jobType").toString(); + if ("workflow.subflow".equals(nodeType)) { + String subFlowName = nodeJsonMap.get("title").toString(); + List DSSFlowList = subflows.stream().filter(subflow -> + subflow.getName().equals(subFlowName) + ).collect(Collectors.toList()); + if (DSSFlowList.size() == 1) { + updateNodeJson = nodeInputService.updateNodeSubflowID(updateNodeJson, DSSFlowList.get(0).getId()); + nodeJsonMap = BDPJettyServerHelper.jacksonJson().readValue(updateNodeJson, Map.class); + nodeJsonListRes.add(nodeJsonMap); + } else if (DSSFlowList.size() > 1) { + logger.error("工程内存在重复的子工作流节点名称，导入失败" + subFlowName); + throw new DSSErrorException(90077, "工程内存在重复的子工作流节点名称，导入失败" + subFlowName); + } else { + logger.error("工程内存在重复的子工作流节点名称，导入失败" + subFlowName); + throw new DSSErrorException(90078, "工程内未能找到子工作流节点，导入失败" + subFlowName); + } + } else { + //todo appconn节点的copy和json更新 + nodeJsonListRes.add(nodeJsonMap); + } + } + } + + return workFlowParser.updateFlowJsonWithKey(flowJson, "nodes", nodeJsonListRes); + } + + + private void persistenceFlowRelation(Long flowID, Long parentFlowID) { + DSSFlowRelation relation = flowMapper.selectFlowRelation(flowID, parentFlowID); + if (relation == null) { + flowMapper.insertFlowRelation(flowID, parentFlowID); + } + } + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/PublishServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/PublishServiceImpl.java new file mode 100644 index 000000000..04390bec8 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/PublishServiceImpl.java @@ -0,0 +1,87 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service.impl; + +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.appconn.scheduler.SchedulerAppConn; +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestFrameworkConvertOrchestration; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.RequestFrameworkConvertOrchestrationStatus; +import com.webank.wedatasphere.dss.orchestrator.common.protocol.ResponseConvertOrchestrator; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.workflow.service.PublishService; +import com.webank.wedatasphere.linkis.rpc.Sender; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class PublishServiceImpl implements PublishService { + + protected final Logger LOGGER = LoggerFactory.getLogger(getClass()); + + protected Sender getOrchestratorSender() { + return DSSSenderServiceFactory.getOrCreateServiceInstance().getOrcSender(); + } + + @Override + public String submitPublish(String convertUser, Long workflowId, + Map dssLabel, Workspace workspace, String comment) throws Exception { + LOGGER.info("User {} begins to convert workflow {}", convertUser, workflowId); + //1 获取对应的orcId 和 orcVersionId + //2.进行提交 + try { + RequestFrameworkConvertOrchestration requestFrameworkConvertOrchestration = new RequestFrameworkConvertOrchestration(); + requestFrameworkConvertOrchestration.setComment(comment); + requestFrameworkConvertOrchestration.setOrcAppId(workflowId); + requestFrameworkConvertOrchestration.setUserName(convertUser); + requestFrameworkConvertOrchestration.setWorkspace(workspace); + SchedulerAppConn schedulerAppConn = AppConnManager.getAppConnManager().getAppConn(SchedulerAppConn.class); + // 只是为了获取是否需要发布所有Orc，这里直接拿第一个AppInstance即可。 + AppInstance appInstance = schedulerAppConn.getAppDesc().getAppInstances().get(0); + requestFrameworkConvertOrchestration.setConvertAllOrcs(schedulerAppConn.getOrCreateWorkflowConversionStandard().getDSSToRelConversionService(appInstance).isConvertAllOrcs()); + requestFrameworkConvertOrchestration.setLabels(dssLabel); + ResponseConvertOrchestrator response = (ResponseConvertOrchestrator) getOrchestratorSender().ask(requestFrameworkConvertOrchestration); + return response.getId(); + } catch (final Exception t) { + LOGGER.error("Failed to submit publish {}.", workflowId, t); + DSSExceptionUtils.dealErrorException(63325, "Failed to submit publish " + workflowId, t, DSSErrorException.class); + } + return null; + } + + @Override + public ResponseConvertOrchestrator getStatus(String username, String taskId) { + if (LOGGER.isDebugEnabled()){ + LOGGER.debug("{} is asking status of {}.", username, taskId); + } + ResponseConvertOrchestrator response =new ResponseConvertOrchestrator(); + + //通过rpc的方式去获取到最新status + try { + RequestFrameworkConvertOrchestrationStatus req = new RequestFrameworkConvertOrchestrationStatus(taskId); + response = (ResponseConvertOrchestrator) getOrchestratorSender().ask(req); + LOGGER.info("user {} gets status of {}, status is {}，msg is {}", username, taskId, response.getResponse().getJobStatus(), response.getResponse().getMessage()); + }catch (Exception t){ + LOGGER.error("failed to getStatus {} ", taskId, t); + + } + return response; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/WorkflowNodeServiceImpl.java b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/WorkflowNodeServiceImpl.java new file mode 100644 index 000000000..8a577a250 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/java/com/webank/wedatasphere/dss/workflow/service/impl/WorkflowNodeServiceImpl.java @@ -0,0 +1,293 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service.impl; + +import com.google.common.collect.Lists; +import com.webank.wedatasphere.dss.appconn.core.AppConn; +import com.webank.wedatasphere.dss.appconn.core.ext.OnlyDevelopmentAppConn; +import com.webank.wedatasphere.dss.appconn.manager.AppConnManager; +import com.webank.wedatasphere.dss.appconn.manager.utils.AppInstanceConstants; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.common.label.EnvDSSLabel; +import com.webank.wedatasphere.dss.common.protocol.project.ProjectRelationRequest; +import com.webank.wedatasphere.dss.common.protocol.project.ProjectRelationResponse; +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import com.webank.wedatasphere.dss.sender.service.DSSSenderServiceFactory; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.ImportRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.NodeRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.OpenRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.service.RefImportService; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AppConnRefFactoryUtils; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; +import com.webank.wedatasphere.dss.workflow.constant.DSSWorkFlowConstant; +import com.webank.wedatasphere.dss.workflow.dao.NodeInfoMapper; +import com.webank.wedatasphere.dss.workflow.entity.AbstractAppConnNode; +import com.webank.wedatasphere.dss.workflow.entity.NodeGroup; +import com.webank.wedatasphere.dss.workflow.entity.NodeInfo; +import com.webank.wedatasphere.dss.workflow.service.WorkflowNodeService; +import com.webank.wedatasphere.linkis.rpc.Sender; +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; +import java.util.List; +import java.util.Map; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +@Service +public class WorkflowNodeServiceImpl implements WorkflowNodeService { + + private Logger logger = LoggerFactory.getLogger(this.getClass()); + + @Autowired + private NodeInfoMapper nodeInfoMapper; + + private Sender projectSender = DSSSenderServiceFactory.getOrCreateServiceInstance().getProjectServerSender(); + + @Override + public List listNodeGroups() { + //cache + return nodeInfoMapper.listNodeGroups(); + } + + + @Override + public Map createNode(String userName, AbstractAppConnNode node) throws ExternalOperationFailedException { + NodeInfo nodeInfo = nodeInfoMapper.getWorkflowNodeByType(node.getNodeType()); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + + DevelopmentIntegrationStandard developmentIntegrationStandard =((OnlyDevelopmentAppConn)appConn).getOrCreateDevelopmentStandard(); + if(null != developmentIntegrationStandard) + { + String label = node.getJobContent().get(DSSCommonUtils.DSS_LABELS_KEY).toString(); + AppInstance appInstance = getAppInstance(appConn, label); + RefCreationOperation refCreationOperation = developmentIntegrationStandard.getRefCRUDService(appInstance).getRefCreationOperation(); + + Workspace workspace = (Workspace) node.getJobContent().get("workspace"); + + NodeRequestRef ref = null; + try { + ref = AppConnRefFactoryUtils + .newAppConnRef(NodeRequestRef.class, refCreationOperation.getClass().getClassLoader(), appConn.getAppDesc().getAppName().toLowerCase()); + } catch (Exception e) { + logger.error("Failed to create CreateNodeRequestRef", e); + } + //todo set create node params + ref.setUserName(userName); + ref.setWorkspace(workspace); + ref.setJobContent(node.getJobContent()); + ref.setName(node.getName()); + if(ref.getName() == null){ + ref.setName(node.getJobContent().get("title").toString()); + } + ref.setOrcId(node.getFlowId()); + ref.setOrcName(node.getFlowName()); + + // parse to external ProjectId + + ref.setProjectId(parseProjectId(node.getProjectId(), appConn.getAppDesc().getAppName(), label)); + ref.setProjectName(node.getProjectName()); + ref.setNodeType(node.getNodeType()); + + ResponseRef responseRef = refCreationOperation.createRef(ref); + return responseRef.toMap(); + } + return null; + } + + private AppInstance getAppInstance(AppConn appConn, String label) { + AppInstance appInstance = null; + for (AppInstance instance : appConn.getAppDesc().getAppInstances()) { + for (DSSLabel dssLabel : instance.getLabels()) { + if(((EnvDSSLabel)dssLabel).getEnv().equalsIgnoreCase(label)){ + appInstance = instance; + break; + } + } + } + return appInstance; + } + + private Long parseProjectId(Long dssProjectId, String appconnName, String labels){ + DSSLabel dssLabel = new EnvDSSLabel(labels); + ProjectRelationRequest projectRelationRequest = new ProjectRelationRequest(dssProjectId, appconnName, Lists.newArrayList(dssLabel)); + ProjectRelationResponse projectRelationResponse = (ProjectRelationResponse) projectSender.ask(projectRelationRequest); + return projectRelationResponse.getAppInstanceProjectId(); + } + + @Override + public void deleteNode(String userName, AbstractAppConnNode node) throws ExternalOperationFailedException { + NodeInfo nodeInfo = nodeInfoMapper.getWorkflowNodeByType(node.getNodeType()); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + DevelopmentIntegrationStandard developmentIntegrationStandard =((OnlyDevelopmentAppConn)appConn).getOrCreateDevelopmentStandard(); + if(null != developmentIntegrationStandard) + { + String label = node.getJobContent().get(DSSCommonUtils.DSS_LABELS_KEY).toString(); + AppInstance appInstance = getAppInstance(appConn, label); + RefDeletionOperation refDeletionOperation = developmentIntegrationStandard.getRefCRUDService(appInstance).getRefDeletionOperation(); + Workspace workspace = (Workspace) node.getJobContent().get("workspace"); + NodeRequestRef ref = null; + try { + ref = AppConnRefFactoryUtils.newAppConnRefByPackageName(NodeRequestRef.class, + appConn.getClass().getClassLoader(), appConn.getClass().getPackage().getName()); + } catch (Exception e) { + logger.error("Failed to create DeleteNodeRequestRef", e); + } + ref.setUserName(userName); + ref.setWorkspace(workspace); + ref.setJobContent(node.getJobContent()); + ref.setName(node.getName()); + ref.setOrcId(node.getFlowId()); + ref.setOrcName(node.getFlowName()); + ref.setProjectId(node.getProjectId()); + ref.setProjectName(node.getProjectName()); + ref.setNodeType(node.getNodeType()); + refDeletionOperation.deleteRef(ref); + } + + } + + @Override + public Map updateNode(String userName, AbstractAppConnNode node) throws ExternalOperationFailedException { + NodeInfo nodeInfo = nodeInfoMapper.getWorkflowNodeByType(node.getNodeType()); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + DevelopmentIntegrationStandard developmentIntegrationStandard =((OnlyDevelopmentAppConn)appConn).getOrCreateDevelopmentStandard(); + if(null != developmentIntegrationStandard) + { + String label = node.getJobContent().get(DSSCommonUtils.DSS_LABELS_KEY).toString(); + AppInstance appInstance = getAppInstance(appConn, label); + RefUpdateOperation refUpdateOperation = developmentIntegrationStandard.getRefCRUDService(appInstance).getRefUpdateOperation(); + Workspace workspace = (Workspace) node.getJobContent().get("workspace"); + NodeRequestRef ref = null; + try { + ref = AppConnRefFactoryUtils.newAppConnRefByPackageName(NodeRequestRef.class, + appConn.getClass().getClassLoader(), appConn.getClass().getPackage().getName()); + } catch (Exception e) { + logger.error("Failed to create UpdateNodeRequestRef", e); + } + ref.setUserName(userName); + ref.setWorkspace(workspace); + ref.setJobContent(node.getJobContent()); + ref.setName(node.getName()); + ref.setOrcId(node.getFlowId()); + ref.setOrcName(node.getFlowName()); + ref.setProjectId(node.getProjectId()); + ref.setProjectName(node.getProjectName()); + ref.setNodeType(node.getNodeType()); + ResponseRef responseRef = refUpdateOperation.updateRef(ref); + return responseRef.toMap(); + }else { + return null; + } + + } + + @Override + public Map refresh(String userName, AbstractAppConnNode node) { + return null; + } + + @Override + public void copyNode(String userName, AbstractAppConnNode newNode, AbstractAppConnNode oldNode) { + + } + + @Override + public void setNodeReadOnly(String userName, AbstractAppConnNode node) { + + } + + @Override + public List listNodes(String userName, AbstractAppConnNode node) { + return null; + } + + @Override + public Map exportNode(String userName, AbstractAppConnNode node) { + return null; + } + + @Override + public Map importNode(String userName, AbstractAppConnNode node, Map resourceMap, + Workspace workspace, String orcVersion) throws Exception { + NodeInfo nodeInfo = nodeInfoMapper.getWorkflowNodeByType(node.getNodeType()); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + EnvDSSLabel dssLabel = new EnvDSSLabel(DSSWorkFlowConstant.DSS_IMPORT_ENV.getValue()); + if (appConn != null) { + DevelopmentIntegrationStandard devStand =((OnlyDevelopmentAppConn)appConn).getOrCreateDevelopmentStandard(); + if (null != devStand) { + + AppInstance appInstance = getAppInstance(appConn, dssLabel.getEnv()); + + RefImportService refImportService = devStand.getRefImportService(appInstance); + ImportRequestRef requestRef = AppConnRefFactoryUtils.newAppConnRef(ImportRequestRef.class, appConn.getClass().getClassLoader(), appConn.getClass().getPackage().getName()); + //todo request param def + requestRef.setParameter("jobContent", node.getJobContent()); + requestRef.setParameter("projectId", parseProjectId(node.getProjectId(), appConn.getAppDesc().getAppName(), dssLabel.getEnv())); + requestRef.setParameter("nodeType", node.getNodeType()); + requestRef.setParameter("orcVersion", orcVersion); + requestRef.setParameter("user", userName); + requestRef.getParameters().putAll(resourceMap); + requestRef.setWorkspace(workspace); + if (null != refImportService) { + ResponseRef responseRef = refImportService.getRefImportOperation().importRef(requestRef); + return responseRef.toMap(); + } + } + } + return null; + } + + @Override + public String getNodeJumpUrl(Map params, AbstractAppConnNode node) throws ExternalOperationFailedException { + NodeInfo nodeInfo = nodeInfoMapper.getWorkflowNodeByType(node.getNodeType()); + AppConn appConn = AppConnManager.getAppConnManager().getAppConn(nodeInfo.getAppConnName()); + DevelopmentIntegrationStandard developmentIntegrationStandard =((OnlyDevelopmentAppConn)appConn).getOrCreateDevelopmentStandard(); + if(null != developmentIntegrationStandard){ + String label = params.get(DSSCommonUtils.DSS_LABELS_KEY).toString(); + AppInstance appInstance = getAppInstance(appConn, label); + RefQueryOperation refQueryOperation = developmentIntegrationStandard.getRefQueryService(appInstance).getRefQueryOperation(); + + String redirectUrl = (String) appInstance.getConfig().get(AppInstanceConstants.REDIRECT_URL); + OpenRequestRef ref = null; + try { + ref = AppConnRefFactoryUtils.newAppConnRefByPackageName(OpenRequestRef.class, appConn.getClass().getClassLoader(), appConn.getClass().getPackage().getName()); + } catch (Exception e) { + logger.error("Failed to create UpdateNodeRequestRef", e); + } + ref.setParameter("params", params); + ref.setParameter("projectId", parseProjectId(node.getProjectId(), appConn.getAppDesc().getAppName(), label)); + try { + ref.setParameter("node", BDPJettyServerHelper.jacksonJson().readValue(BDPJettyServerHelper.jacksonJson().writeValueAsString(node), Map.class)); + } catch (Exception e) { + logger.error("Failed to convert node to map", e); + } + ref.setParameter("redirectUrl", redirectUrl); + ResponseRef responseRef = refQueryOperation.query(ref); + return responseRef.getValue("jumpUrl").toString(); + } + return null; + } +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/DSSWorkflowServerApplication.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/DSSWorkflowServerApplication.scala new file mode 100644 index 000000000..e18e1b260 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/DSSWorkflowServerApplication.scala @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow + +import com.webank.wedatasphere.dss.common.utils.DSSMainHelper +import com.webank.wedatasphere.linkis.DataWorkCloudApplication +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} + + +object DSSWorkflowServerApplication extends Logging { + + val userName: String = System.getProperty("user.name") + val hostName: String = Utils.getComputerName + + def main(args: Array[String]): Unit = { + val serviceName = System.getProperty("serviceName")//ProjectConf.SERVICE_NAME.getValue + DSSMainHelper.formatPropertyFiles(serviceName) + val allArgs = args ++ DSSMainHelper.getExtraSpringOptions + System.setProperty("hostName", hostName) + System.setProperty("userName", userName) + info(s"Ready to start $serviceName with args: ${allArgs.toList}.") + println(s"Test Ready to start $serviceName with args: ${allArgs.toList}.") + DataWorkCloudApplication.main(allArgs) + } +} \ No newline at end of file diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/receiver/DSSWorkflowReceiver.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/receiver/DSSWorkflowReceiver.scala new file mode 100644 index 000000000..fd8bc5cb7 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/receiver/DSSWorkflowReceiver.scala @@ -0,0 +1,120 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.receiver + + +import java.util + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException +import com.webank.wedatasphere.dss.common.protocol._ +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils +import com.webank.wedatasphere.dss.orchestrator.common.protocol.{RequestConvertOrchestrations, ResponseOperateOrchestrator} +import com.webank.wedatasphere.dss.standard.app.sso.Workspace +import com.webank.wedatasphere.dss.workflow.WorkFlowManager +import com.webank.wedatasphere.dss.workflow.common.entity.DSSFlow +import com.webank.wedatasphere.dss.workflow.common.protocol._ +import com.webank.wedatasphere.dss.workflow.entity.DSSFlowImportParam +import com.webank.wedatasphere.linkis.rpc.{Receiver, Sender} +import org.springframework.beans.factory.annotation.Autowired +import org.springframework.stereotype.Component + +import scala.concurrent.duration.Duration + + +@Component +class DSSWorkflowReceiver extends Receiver { + + @Autowired + var workflowManager: WorkFlowManager = _ + + + override def receive(message: Any, sender: Sender): Unit = {} + + override def receiveAndReply(message: Any, sender: Sender): Any = message match { + + case reqCreateFlow: RequestCreateWorkflow => + val dssFlow: DSSFlow = workflowManager.createWorkflow(reqCreateFlow.getUserName, reqCreateFlow.getWorkflowName, + reqCreateFlow.getContextIDStr, reqCreateFlow.getDescription, + reqCreateFlow.getParentFlowID, reqCreateFlow.getUses, + reqCreateFlow.getLinkedAppConnNames, reqCreateFlow.getDssLabels) + val responseCreateWorkflow = new ResponseCreateWorkflow() + responseCreateWorkflow.setDssFlow(dssFlow) + responseCreateWorkflow + + case reqUpdateFlow: RequestUpdateWorkflow => + workflowManager.updateWorkflow(reqUpdateFlow.userName, + reqUpdateFlow.flowID, reqUpdateFlow.flowName, + reqUpdateFlow.description, reqUpdateFlow.uses) + new ResponseUpdateWorkflow(JobStatus.Success) + + case reqDeleteFlow: RequestDeleteWorkflow => + workflowManager.deleteWorkflow(reqDeleteFlow.userName, reqDeleteFlow.flowID) + new ResponseDeleteWorkflow(JobStatus.Success) + + case reqExportFlow: RequestExportWorkflow => + val dssExportFlowResource: util.Map[String, AnyRef] = workflowManager.exportWorkflow( + reqExportFlow.userName, + reqExportFlow.flowID, + reqExportFlow.projectId, + reqExportFlow.projectName, + DSSCommonUtils.COMMON_GSON.fromJson(reqExportFlow.workspaceStr, classOf[Workspace]), + reqExportFlow.dssLabelList) + ResponseExportWorkflow(dssExportFlowResource.get("resourceId").toString, dssExportFlowResource.get("version").toString, + reqExportFlow.flowID) + + case requestImportWorkflow: RequestImportWorkflow => + val dssFlowImportParam: DSSFlowImportParam = new DSSFlowImportParam() + dssFlowImportParam.setProjectID(requestImportWorkflow.getProjectId) + dssFlowImportParam.setProjectName(requestImportWorkflow.getProjectName) + dssFlowImportParam.setSourceEnv(requestImportWorkflow.getSourceEnv) + dssFlowImportParam.setUserName(requestImportWorkflow.getUserName) + dssFlowImportParam.setVersion(requestImportWorkflow.getBmlVersion) + dssFlowImportParam.setOrcVersion(requestImportWorkflow.getOrcVersion) + dssFlowImportParam.setWorkspaceName(requestImportWorkflow.getWorkspaceName) + dssFlowImportParam.setWorkspace(DSSCommonUtils.COMMON_GSON.fromJson(requestImportWorkflow.getWorkspaceStr, classOf[Workspace])) + dssFlowImportParam.setContextId(requestImportWorkflow.getContextId) + val dssFlows = workflowManager.importWorkflow(requestImportWorkflow.getUserName, + requestImportWorkflow.getResourceId, + requestImportWorkflow.getBmlVersion, + dssFlowImportParam) + import scala.collection.JavaConversions._ + val dssFlowIds = dssFlows.map(dssFlow => dssFlow.getId).toList + new ResponseImportWorkflow(JobStatus.Success, dssFlowIds) + + case requestCopyWorkflow: RequestCopyWorkflow => + val copyFlow: DSSFlow = workflowManager.copyRootflowWithSubflows(requestCopyWorkflow.getUserName, + requestCopyWorkflow.getRootFlowId, + requestCopyWorkflow.getWorkspaceName, + requestCopyWorkflow.getProjectName, + requestCopyWorkflow.getContextIdStr, + requestCopyWorkflow.getVersion, + requestCopyWorkflow.getDescription) + new ResponseCopyWorkflow(copyFlow) + + case requestQueryWorkFlow: RequestQueryWorkFlow => + val dssFlow: DSSFlow = workflowManager.queryWorkflow(requestQueryWorkFlow.userName, + requestQueryWorkFlow.rootFlowId) + new ResponseQueryWorkflow(dssFlow) + + case requestConvertOrchestrator: RequestConvertOrchestrations => + workflowManager.convertWorkflow(requestConvertOrchestrator) + + case _ => throw new DSSErrorException(90000, "Not support protocol " + message) + } + + override def receiveAndReply(message: Any, duration: Duration, sender: Sender): Any = {} +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/service/BMLService.scala b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/service/BMLService.scala new file mode 100644 index 000000000..6f70a41c1 --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/dss-workflow-server/src/main/scala/com/webank/wedatasphere/dss/workflow/service/BMLService.scala @@ -0,0 +1,201 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.workflow.service + +import java.io.{ByteArrayInputStream, InputStream} +import java.util +import java.util.UUID + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException +import com.webank.wedatasphere.dss.common.utils.IoUtils +import com.webank.wedatasphere.linkis.bml.client.{BmlClient, BmlClientFactory} +import com.webank.wedatasphere.linkis.bml.protocol.{BmlDownloadResponse, BmlUpdateResponse, BmlUploadResponse} +import com.webank.wedatasphere.linkis.common.utils.{JavaLog, Utils} +import org.apache.commons.io.IOUtils +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ + + +@Component +class BMLService extends JavaLog{ + + def upload(userName: String, content: String, fileName: String, projectName:String): util.Map[String, Object] = { + val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) + val client: BmlClient = createBMLClient(userName) + val resource: BmlUploadResponse = client.uploadShareResource(userName, projectName, fileName, inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def upload(userName: String, inputStream: InputStream, fileName: String, projectName:String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + val resource: BmlUploadResponse = client.uploadShareResource(userName, projectName, fileName, inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def update(userName: String, resourceId: String, inputStream: InputStream): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + val resource: BmlUpdateResponse = client.updateShareResource(userName, resourceId, "", inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def update(userName: String, resourceId: String, content: String): util.Map[String, Object] = { + val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) + val client: BmlClient = createBMLClient(userName) + val resource: BmlUpdateResponse = client.updateShareResource(userName, resourceId, UUID.randomUUID().toString+".json", inputStream) + if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") + val map = new util.HashMap[String, Object] + map += "resourceId" -> resource.resourceId + map += "version" -> resource.version + } + + def query(userName: String, resourceId: String, version: String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + var resource: BmlDownloadResponse = null + if (version == null) { + resource = client.downloadShareResource(userName, resourceId) + } else { + resource = client.downloadShareResource(userName, resourceId, version) + } + if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") + val map = new util.HashMap[String, Object] + map += "path" -> resource.fullFilePath + map += "string" -> inputstremToString(resource.inputStream) + } + + def download(userName: String, resourceId: String, version: String): util.Map[String, Object] = { + val client: BmlClient = createBMLClient(userName) + var resource: BmlDownloadResponse = null + if (version == null) { + resource = client.downloadShareResource(userName, resourceId) + } else { + resource = client.downloadShareResource(userName, resourceId, version) + } + if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") + val map = new util.HashMap[String, Object] + map += "path" -> resource.fullFilePath + map += "is" -> resource.inputStream + } + + def downloadToLocalPath(userName: String, resourceId: String, version: String, path: String): String = { + val result = download(userName,resourceId,version) + val is = result.get("is").asInstanceOf[InputStream] + val os = IoUtils.generateExportOutputStream(path) + Utils.tryFinally(IOUtils.copy(is,os)){ + IOUtils.closeQuietly(os) + IOUtils.closeQuietly(is) + } + path + } + + def downloadAndGetFlowJson(userName: String, resourceId: String, version: String, path: String): String = { + downloadToLocalPath(userName,resourceId,version,path) + //因为下载到指定目录后返回的resource的Stream为null,只能从文件重新读取。 + val is = IoUtils.generateInputInputStream(path) + Utils.tryFinally(inputstremToString(is))(IOUtils.closeQuietly(is)) + } + + def readLocalResourceFile(userName: String,readPath: String): InputStream ={ + IoUtils.generateInputInputStream(readPath) + } + + def readLocalFlowJsonFile(userName: String,readPath: String): String ={ + var inputStream:InputStream = null + var flowJson:String = null + Utils.tryFinally{ + inputStream = IoUtils.generateInputInputStream(readPath) + flowJson = inputstremToString(inputStream) + }{ + IOUtils.closeQuietly(inputStream) + } + flowJson + } + + + def readFlowJsonFromBML(userName: String, resourceId:String ,version: String ): String ={ + + val result = download(userName,resourceId,version) + val is = result.get("is").asInstanceOf[InputStream] + + var inputStream:InputStream = null + var flowJson:String = null + Utils.tryFinally{ + inputStream =is + flowJson = inputstremToString(inputStream) + }{ + IOUtils.closeQuietly(inputStream) + } + flowJson + } + + + private def inputstremToString(inputStream: InputStream): String = { + scala.io.Source.fromInputStream(inputStream).mkString + } + + private def createBMLClient(userName: String): BmlClient = { + if (userName == null) + BmlClientFactory.createBmlClient() + else + BmlClientFactory.createBmlClient(userName) + } + + def createBmlProject(username:String, projectName:String, editUsers:util.List[String], + accessUsers:util.List[String] ): Unit ={ + val client = createBMLClient(username) + val response = client.createBmlProject(username, projectName, accessUsers, editUsers) + if (response.isSuccess){ + logger.info(s"for user $username create bml project $projectName success") + }else{ + logger.error(s"for user $username create bml project $projectName failed") + } + } + + def attachResourceAndProject(username:String, projectName:String, resourceId:String):Unit = { + val client = createBMLClient(username) + val response = client.attachResourceAndProject(projectName, resourceId) + if (response.isSuccess){ + logger.info(s"attach $username and $projectName success") + }else{ + logger.error(s"attach $username and $projectName failed") + } + } + + + def updateProjectPriv(projectName:String, username:String, editUsers:util.List[String], + accessUsers:util.List[String]): Unit ={ + val client = createBMLClient(username) + val response = client.updateProjectPriv(username, projectName, editUsers, accessUsers) + if (response.isSuccess){ + logger.info(s"attach $username and $projectName success") + }else{ + logger.error(s"attach $username and $projectName failed") + } + } + + + +} diff --git a/dss-orchestrator/orchestrators/dss-workflow/pom.xml b/dss-orchestrator/orchestrators/dss-workflow/pom.xml new file mode 100644 index 000000000..86b2505fa --- /dev/null +++ b/dss-orchestrator/orchestrators/dss-workflow/pom.xml @@ -0,0 +1,40 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-workflow + pom + + + dss-workflow-common + dss-linkis-node-execution + dss-flow-execution-server + dss-workflow-server + dss-workflow-sdk + dss-workflow-conversion-standard + + + \ No newline at end of file diff --git a/dss-orchestrator/pom.xml b/dss-orchestrator/pom.xml new file mode 100644 index 000000000..ecd8caf1a --- /dev/null +++ b/dss-orchestrator/pom.xml @@ -0,0 +1,40 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-orchestrator + pom + + + dss-orchestrator-common + dss-orchestrator-core + dss-orchestrator-db + dss-orchestrator-loader + orchestrators/dss-workflow + dss-orchestrator-conversion-standard + + + diff --git a/dss-scheduler-appjoint-core/pom.xml b/dss-scheduler-appjoint-core/pom.xml deleted file mode 100644 index 72f112372..000000000 --- a/dss-scheduler-appjoint-core/pom.xml +++ /dev/null @@ -1,76 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-scheduler-appjoint-core - - - org.springframework - spring-beans - 5.0.7.RELEASE - - - com.webank.wedatasphere.dss - dss-appjoint-core - - - linkis-protocol - com.webank.wedatasphere.linkis - - - linkis-storage - com.webank.wedatasphere.linkis - - - ${dss.version} - - - com.google.code.gson - gson - 2.8.5 - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - - - \ No newline at end of file diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/SchedulerAppJoint.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/SchedulerAppJoint.java deleted file mode 100644 index 9923d90c7..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/SchedulerAppJoint.java +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.ProjectPublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.ProjectParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.service.SchedulerProjectService; -import com.webank.wedatasphere.dss.appjoint.scheduler.service.SchedulerSecurityService; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.ProjectTuning; - -import java.io.Closeable; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface SchedulerAppJoint extends AppJoint, Closeable { - - SchedulerProjectService getProjectService(); - - SchedulerSecurityService getSecurityService(); - - ProjectParser getProjectParser(); - ProjectTuning getProjectTuning(); - - ProjectPublishHook[] getProjectPublishHooks(); - - -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/constant/SchedulerAppJointConstant.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/constant/SchedulerAppJointConstant.java deleted file mode 100644 index 25d4e9d84..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/constant/SchedulerAppJointConstant.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.constant; - -/** - * Created by enjoyyin on 2019/11/5. - */ -public class SchedulerAppJointConstant { - public final static String PROXY_USER = "user.to.proxy"; - public final static String CONFIGURATION = "configuration"; - public final static String RUNTIME = "runtime"; - public final static String CATEGORY = "category"; - public final static String SENDEMAIL_NODE_TYPE = "linkis.appjoint.sendemail"; - public final static String NODE = "node"; -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/AbstractSchedulerNode.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/AbstractSchedulerNode.java deleted file mode 100644 index a5139d7d9..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/AbstractSchedulerNode.java +++ /dev/null @@ -1,90 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; - -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractSchedulerNode implements SchedulerNode { - - private DSSNode dssNode; - - @Override - public DSSNode getDssNode() { - return this.dssNode; - } - - @Override - public void setDssNode(DSSNode dssNode) { - this.dssNode = dssNode; - } - - @Override - public String getId() { - return dssNode.getId(); - } - - @Override - public void setId(String id) { - dssNode.setId(id); - } - - @Override - public String getNodeType() { - return dssNode.getNodeType(); - } - - @Override - public void setNodeType(String nodeType) { - dssNode.setNodeType(nodeType); - } - - @Override - public String getName() { - return dssNode.getName(); - } - - @Override - public void setName(String name) { - dssNode.setName(name); - } - - @Override - public void addDependency(String nodeName) { - dssNode.addDependency(nodeName); - } - - @Override - public void setDependency(List dependency) { - dssNode.setDependency(dependency); - } - - @Override - public void removeDependency(String nodeName) { - dssNode.removeDependency(nodeName); - } - - @Override - public List getDependencys() { - return dssNode.getDependencys(); - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/AbstractSchedulerProject.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/AbstractSchedulerProject.java deleted file mode 100644 index 3a8ad55eb..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/AbstractSchedulerProject.java +++ /dev/null @@ -1,112 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.ProjectVersion; - -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractSchedulerProject implements SchedulerProject { - - private Long id; - private String name; - private String description; - - private DSSProject dssProject; - private List schedulerFlows; - - private List projectVersions; - - public List getSchedulerFlows() { - return schedulerFlows; - } - - public void setSchedulerFlows(List schedulerFlows) { - this.schedulerFlows = schedulerFlows; - } - - @Override - public Long getId() { - return this.id; - } - - @Override - public void setId(Long id) { - this.id = id; - } - - @Override - public String getName() { - return this.name; - } - - @Override - public void setName(String name) { - this.name = name; - } - - @Override - public String getDescription() { - return this.description; - } - - @Override - public void setDescription(String description) { - this.description = description; - } - - @Override - public String getPorjectGroup() { - return null; - } - - @Override - public void setProjectGroup(String projectGroup) { - - } - - @Override - public List getProjectVersions() { - return this.projectVersions; - } - - @Override - public void setProjectVersions(List projectVersions) { - this.projectVersions = projectVersions.stream().map(f ->(SchedulerProjectVersionForFlows)f).collect(Collectors.toList()); - } - - @Override - public void addProjectVersion(ProjectVersion projectVersion) { - this.projectVersions.add((SchedulerProjectVersionForFlows)projectVersion); - } - - @Override - public DSSProject getDssProject() { - return this.dssProject; - } - - @Override - public void setDssProject(DSSProject dssProject) { - this.dssProject = dssProject; - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/FileProjectVersion.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/FileProjectVersion.java deleted file mode 100644 index 0a09f6081..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/FileProjectVersion.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.project.ProjectVersion; - -import java.io.File; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface FileProjectVersion extends ProjectVersion { - - File getProjectFile(); - - void setProjectFile(); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/ReadNode.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/ReadNode.java deleted file mode 100644 index 90f6951e6..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/ReadNode.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface ReadNode extends SchedulerNode { - String[] getShareNodeIds(); - void setShareNodeIds(String[] nodeIds); - SchedulerNode getSchedulerNode(); - void setSchedulerNode(SchedulerNode schedulerNode); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerEdge.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerEdge.java deleted file mode 100644 index aee1e7229..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerEdge.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.node.DSSEdge; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface SchedulerEdge { - DSSEdge getDssEdge(); - - void setDssEdge(DSSEdge dssEdge); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerEdgeDefault.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerEdgeDefault.java deleted file mode 100644 index db663fe54..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerEdgeDefault.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.node.DSSEdge; - -/** - * Created by allenlliu on 2019/9/19. - */ -public class SchedulerEdgeDefault implements SchedulerEdge { - private DSSEdge dssEdge; - - @Override - public DSSEdge getDssEdge() { - return dssEdge; - } - - @Override - public void setDssEdge(DSSEdge dssEdge) { - this.dssEdge = dssEdge; - } - -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerFlow.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerFlow.java deleted file mode 100644 index e3cd2a542..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerFlow.java +++ /dev/null @@ -1,161 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.Resource; -import com.webank.wedatasphere.dss.common.entity.flow.Flow; -import com.webank.wedatasphere.dss.common.entity.flow.FlowVersion; - -import java.util.Date; -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public class SchedulerFlow implements Flow { - private Long id; - private String name; - private String description; - private String type; - private List children; - private Boolean isRootFlow; - private String userProxy; - private List schedulerNodes; - private List schedulerEdges; - private List flowResources; - private List> flowProperties; - - public String getUserProxy() { - return userProxy; - } - - public void setUserProxy(String userProxy) { - this.userProxy = userProxy; - } - - public List getSchedulerNodes() { - return schedulerNodes; - } - - public void setSchedulerNodes(List schedulerNodes) { - this.schedulerNodes = schedulerNodes; - } - - public List getSchedulerEdges() { - return schedulerEdges; - } - - public void setSchedulerEdges(List schedulerEdges) { - this.schedulerEdges = schedulerEdges; - } - - public List getFlowResources() { - return flowResources; - } - - public void setFlowResources(List flowResources) { - this.flowResources = flowResources; - } - - public List> getFlowProperties() { - return flowProperties; - } - - public void setFlowProperties(List> flowProperties) { - this.flowProperties = flowProperties; - } - - @Override - public Long getId() { - return this.id; - } - - @Override - public void setId(Long id) { - this.id = id; - } - - @Override - public String getName() { - return this.name; - } - - @Override - public void setName(String name) { - this.name = name; - } - - @Override - public String getFlowType() { - return this.type; - } - - @Override - public void setFlowType(String flowType) { - this.type = flowType; - - } - - @Override - public String getDescription() { - return this.description; - } - - @Override - public void setDescription(String description) { - this.description = description; - } - - @Override - public Boolean getRootFlow() { - return this.isRootFlow; - } - - public void setRootFlow(Boolean rootFlow) { - isRootFlow = rootFlow; - } - - @Override - public List getChildren() { - return this.children; - } - - @Override - public void setChildren(List children) { - this.children = children.stream().map(f ->(SchedulerFlow)f).collect(Collectors.toList()); - } - - - @Override - public List getFlowVersions() { - return null; - } - - @Override - public void setFlowVersions(List flowVersions) { - - } - - - @Override - public void addFlowVersion(FlowVersion flowVersion) { - - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerNode.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerNode.java deleted file mode 100644 index 612c2e25d..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerNode.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - - -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.common.entity.node.Node; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface SchedulerNode extends Node { - DSSNode getDssNode(); - - void setDssNode(DSSNode dssNode); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerProject.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerProject.java deleted file mode 100644 index d4158b0f4..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerProject.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.Project; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface SchedulerProject extends Project { - DSSProject getDssProject(); - void setDssProject(DSSProject dssProject); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerProjectVersionForFlows.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerProjectVersionForFlows.java deleted file mode 100644 index bc915cb47..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/SchedulerProjectVersionForFlows.java +++ /dev/null @@ -1,73 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -import com.webank.wedatasphere.dss.common.entity.flow.Flow; -import com.webank.wedatasphere.dss.common.entity.project.ProjectVersionForFlows; - -import java.util.ArrayList; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/26. - */ -public class SchedulerProjectVersionForFlows implements ProjectVersionForFlows { - - private List allSchedulerFlows = new ArrayList<>(); - - @Override - public String getVersion() { - return null; - } - - @Override - public void setVersion() { - - } - - @Override - public String getComment() { - return null; - } - - @Override - public void setComment(String comment) { - - } - - @Override - public List getFlows() { - return this.allSchedulerFlows; - } - - @Override - public void setFlows(List flows) { - this.allSchedulerFlows = flows.stream().map(f ->(SchedulerFlow)f).collect(Collectors.toList()); - } - - @Override - public void addFlow(Flow flow) { - this.allSchedulerFlows.add((SchedulerFlow) flow); - } - - @Override - public void removeFlow(Flow flow) { - - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/ShareNode.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/ShareNode.java deleted file mode 100644 index 41a0bb494..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/entity/ShareNode.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.entity; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface ShareNode extends SchedulerNode { - int getShareTimes(); - void setShareTimes(int num); - SchedulerNode getSchedulerNode(); - void setSchedulerNode(SchedulerNode node); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractFlowPublishHook.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractFlowPublishHook.java deleted file mode 100644 index dcb2f2a49..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractFlowPublishHook.java +++ /dev/null @@ -1,64 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.hooks; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; - -import java.util.Arrays; -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractFlowPublishHook implements FlowPublishHook { - - private NodePublishHook[] nodePublishHooks; - - @Override - public void setNodeHooks(NodePublishHook[] nodePublishHooks) { - this.nodePublishHooks = nodePublishHooks; - } - - @Override - public NodePublishHook[] getNodeHooks() { - return this.nodePublishHooks; - } - - @Override - public void prePublish(SchedulerFlow flow) throws DSSErrorException { - // 调用nodeHook - List schedulerNodes = flow.getSchedulerNodes(); - Arrays.stream(getNodeHooks()).forEach(h -> invokePreNodePublishHook(schedulerNodes,h)); - //递归调用自身方法 - if(flow.getChildren() !=null){ - flow.getChildren().forEach(DSSExceptionUtils.handling(this::prePublish)); - } - } - - @Override - public void postPublish(SchedulerFlow flow) { - // TODO: 2019/9/25 递归调用nodepublishHook - } - - private void invokePreNodePublishHook(List schedulerNodes, NodePublishHook nodePublishHook){ - schedulerNodes.forEach(DSSExceptionUtils.handling(nodePublishHook::prePublish)); - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractNodePublishHook.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractNodePublishHook.java deleted file mode 100644 index fd43549c5..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractNodePublishHook.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.hooks; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractNodePublishHook implements NodePublishHook { - @Override - public void prePublish(SchedulerNode schedulerNode) throws DSSErrorException { - //默认的不做处理 - } - - @Override - public void postPublish(SchedulerNode schedulerNode) { - //默认的不做处理 - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractProjectPublishHook.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractProjectPublishHook.java deleted file mode 100644 index 5d059cd1f..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/AbstractProjectPublishHook.java +++ /dev/null @@ -1,64 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.hooks; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; - -import java.util.Arrays; -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractProjectPublishHook implements ProjectPublishHook { - - @Override - public FlowPublishHook[] getFlowPublishHooks() { - return flowPublishHooks; - } - - @Override - public void setFlowPublishHooks(FlowPublishHook[] flowPublishHooks) { - this.flowPublishHooks = flowPublishHooks; - } - - private FlowPublishHook[] flowPublishHooks; - - @Override - public void prePublish(SchedulerProject project) throws DSSErrorException { - //这里强转也是因为SchedulerProject ， - //如果azkaban有实现，应该要继承AbstractSchedulerProject 而不是直接继承SchedulerProject - //调用flowHook的prePublish - AbstractSchedulerProject schedulerProject = (AbstractSchedulerProject) project; - List schedulerFlows = schedulerProject.getSchedulerFlows(); - Arrays.stream(getFlowPublishHooks()).forEach(h -> invokePreFlowPublishHook(schedulerFlows,h)); - } - - @Override - public void postPublish(SchedulerProject project) throws DSSErrorException { - //// TODO: 2019/9/25 递归调用flowHooks 的post - } - - private void invokePreFlowPublishHook(List schedulerFlows,FlowPublishHook flowPublishHook){ - schedulerFlows.forEach(DSSExceptionUtils.handling(flowPublishHook::prePublish)); - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/FlowPublishHook.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/FlowPublishHook.java deleted file mode 100644 index e75420de7..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/FlowPublishHook.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.hooks; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - -/** - * Created by enjoyyin on 2019/11/7. - */ -public interface FlowPublishHook { - void prePublish(SchedulerFlow flow) throws DSSErrorException; - void postPublish(SchedulerFlow flow)throws DSSErrorException; - void setNodeHooks(NodePublishHook[] nodePublishHooks); - NodePublishHook[] getNodeHooks(); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/NodePublishHook.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/NodePublishHook.java deleted file mode 100644 index 606dbd171..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/NodePublishHook.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.hooks; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface NodePublishHook { - void prePublish(SchedulerNode schedulerNode) throws DSSErrorException; - void postPublish(SchedulerNode schedulerNode)throws DSSErrorException; -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/ProjectPublishHook.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/ProjectPublishHook.java deleted file mode 100644 index aac909c5a..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/hooks/ProjectPublishHook.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.hooks; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - -/** - * Created by enjoyyin on 2019/9/18 - */ -public interface ProjectPublishHook { - void prePublish(SchedulerProject project) throws DSSErrorException; - void postPublish(SchedulerProject project) throws DSSErrorException; - FlowPublishHook[] getFlowPublishHooks(); - void setFlowPublishHooks(FlowPublishHook[] flowPublishHooks); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/order/Order.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/order/Order.java deleted file mode 100644 index 8baf3d0a4..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/order/Order.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.order; - -/** - * order接口用于标识parser和tunning，hook的调用顺序 - * Created by enjoyyin on 2019/9/30. - */ -public interface Order { - - int getOrder(); - -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractFlowParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractFlowParser.java deleted file mode 100644 index 96a9f2aed..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractFlowParser.java +++ /dev/null @@ -1,117 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.google.gson.Gson; -import com.google.gson.JsonArray; -import com.google.gson.JsonObject; -import com.google.gson.JsonParser; -import com.google.gson.reflect.TypeToken; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerEdge; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerEdgeDefault; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.common.entity.Resource; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSJSONFlow; -import com.webank.wedatasphere.dss.common.entity.flow.Flow; -import com.webank.wedatasphere.dss.common.entity.node.DSSEdge; -import com.webank.wedatasphere.dss.common.entity.node.DSSEdgeDefault; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.common.entity.node.DSSNodeDefault; -import org.springframework.beans.BeanUtils; - -import java.util.*; - - -/** - * Created by enjoyyin on 2019/9/7. - */ -public abstract class AbstractFlowParser implements FlowParser { - - private NodeParser[] nodeParsers; - - @Override - public void setNodeParsers(NodeParser[] nodeParsers) { - this.nodeParsers = nodeParsers; - } - - @Override - public NodeParser[] getNodeParsers() { - return nodeParsers; - } - - //留白 - protected void downloadFlowResources(){} - - //留白 - protected void dealFlowResources(){} - - protected void dealFlowProperties(Flow flow){} - - @Override - public SchedulerFlow parseFlow(DSSJSONFlow flow) { - downloadFlowResources(); - dealFlowResources(); - dealFlowProperties(flow); - return resolveDSSJSONFlow(flow); - } - - // 解析DSSJSONFlow，生成DSSNode - public SchedulerFlow resolveDSSJSONFlow(DSSJSONFlow jsonFlow){ - SchedulerFlow schedulerFlow = createSchedulerFlow(); - BeanUtils.copyProperties(jsonFlow,schedulerFlow,"children"); - JsonParser parser = new JsonParser(); - JsonObject jsonObject = parser.parse(jsonFlow.getJson()).getAsJsonObject(); - JsonArray nodeJsonArray = jsonObject.getAsJsonArray("nodes"); - Gson gson = new Gson(); - List dssNodes = gson.fromJson(nodeJsonArray, new TypeToken>() { - }.getType()); - List schedulerNodeList = new ArrayList<>(); - List schedulerEdgeList = new ArrayList<>(); - for (DSSNode dssNode : dssNodes) { - Optional firstNodeParser = Arrays.stream(getNodeParsers()) - .filter(p -> p.ifNodeCanParse(dssNode)) - .sorted((p1, p2) -> p2.getOrder() - p1.getOrder()) - .findFirst(); - SchedulerNode schedulerNode = firstNodeParser.orElseThrow(()->new IllegalArgumentException("NodeParser个数应该大于0")).parseNode(dssNode); - schedulerNodeList.add(schedulerNode); - } - JsonArray edgeJsonArray = jsonObject.getAsJsonArray("edges"); - List dssEdges = gson.fromJson(edgeJsonArray, new TypeToken>() { - }.getType()); - for (DSSEdge dssEdge : dssEdges) { - SchedulerEdge schedulerEdge = new SchedulerEdgeDefault(); - schedulerEdge.setDssEdge(dssEdge); - schedulerEdgeList.add(schedulerEdge); - - } - JsonArray proJsonArray = jsonObject.getAsJsonArray("props"); - List> props = gson.fromJson(proJsonArray, new TypeToken>>() { - }.getType()); - JsonArray resourcesJsonArray = jsonObject.getAsJsonArray("resources"); - List resources = gson.fromJson(resourcesJsonArray, new TypeToken>() { - }.getType()); - schedulerFlow.setFlowResources(resources); - schedulerFlow.setFlowProperties(props); - schedulerFlow.setSchedulerEdges(schedulerEdgeList); - schedulerFlow.setSchedulerNodes(schedulerNodeList); - return schedulerFlow; - } - - protected SchedulerFlow createSchedulerFlow(){return new SchedulerFlow();} -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractNodeParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractNodeParser.java deleted file mode 100644 index c12f0ab76..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractNodeParser.java +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerEdge; - -import java.util.List; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public abstract class AbstractNodeParser implements NodeParser { - //留白 - protected Boolean downloadNodeResource(){return null;}; - - //留白 - protected Boolean downloadNodeContent(){return null;}; - - //留白 - protected void getNodeVariables(){}; - - protected List resolveDependencys(SchedulerNode node, List flowEdges){return null;}; - -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractProjectParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractProjectParser.java deleted file mode 100644 index a7d481d24..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractProjectParser.java +++ /dev/null @@ -1,105 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProjectVersionForFlows; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSJSONFlow; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.DSSJSONProject; -import com.webank.wedatasphere.dss.common.entity.project.ProjectVersionForFlows; -import org.springframework.beans.BeanUtils; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public abstract class AbstractProjectParser implements ProjectParser { - - private FlowParser[] flowParsers; - - @Override - public void setFlowParsers(FlowParser[] flowParsers) { - this.flowParsers = flowParsers; - } - - @Override - public FlowParser[] getFlowParsers() { - return flowParsers; - } - - public DSSJSONProject parseToDssJsonProject(DSSProject dssProject){ - DSSJSONProject dssJsonProject = new DSSJSONProject(); - BeanUtils.copyProperties(dssProject, dssJsonProject,"flows","projectVersions"); - List dwsFlows = dssProject.getFlows(); - List dssJsonFlows = dwsFlows.stream().map(this::toDssJsonFlow).collect(Collectors.toList()); - dssJsonProject.setFlows(dssJsonFlows); - return dssJsonProject; - } - - private DSSJSONFlow toDssJsonFlow(DSSFlow dssFlow){ - DSSJSONFlow dssJsonFlow = new DSSJSONFlow(); - BeanUtils.copyProperties(dssFlow, dssJsonFlow,"children","flowVersions"); - dssJsonFlow.setJson(dssFlow.getLatestVersion().getJson()); - if(dssFlow.getChildren() != null){ - dssJsonFlow.setChildren(dssFlow.getChildren().stream().map(this::toDssJsonFlow).collect(Collectors.toList())); - } - return dssJsonFlow; - } - - - public SchedulerProject parseProject(DSSJSONProject project){ - AbstractSchedulerProject schedulerProject = createSchedulerProject(); - SchedulerProjectVersionForFlows projectVersionForFlows = new SchedulerProjectVersionForFlows(); - schedulerProject.setProjectVersions(new ArrayList()); - schedulerProject.addProjectVersion(projectVersionForFlows); - BeanUtils.copyProperties(project,schedulerProject,"projectVersions"); - List schedulerFlows = project.getFlows().stream().map(f -> invokeFlowParser(projectVersionForFlows,f, getFlowParsers())).collect(Collectors.toList()); - schedulerProject.setSchedulerFlows(schedulerFlows); - return schedulerProject; - } - - private SchedulerFlow invokeFlowParser(ProjectVersionForFlows projectVersionForFlows, DSSJSONFlow dssJsonFlow, FlowParser[] flowParsers){ - List flowParsersF = Arrays.stream(flowParsers).filter(f -> f.ifFlowCanParse(dssJsonFlow)).collect(Collectors.toList()); - // TODO: 2019/9/25 如果flowParsers数量>1 ||<=0抛出异常 - SchedulerFlow schedulerFlow = flowParsersF.get(0).parseFlow(dssJsonFlow); - //收集所有的不分层级的flow？ - projectVersionForFlows.addFlow(schedulerFlow); - if(dssJsonFlow.getChildren() != null){ - List schedulerFlows = dssJsonFlow.getChildren().stream().map(f -> invokeFlowParser(projectVersionForFlows,f, flowParsers)).collect(Collectors.toList()); - schedulerFlow.setChildren(schedulerFlows); - } - return schedulerFlow; - } - - @Override - public SchedulerProject parseProject(DSSProject dssProject) { - SchedulerProject schedulerProject = parseProject(parseToDssJsonProject(dssProject)); - schedulerProject.setDssProject(dssProject); - return schedulerProject; - } - - protected abstract AbstractSchedulerProject createSchedulerProject(); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractReadNodeParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractReadNodeParser.java deleted file mode 100644 index 68e8d970e..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/AbstractReadNodeParser.java +++ /dev/null @@ -1,77 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; - -import java.util.Arrays; -import java.util.Map; - -/** - * Created by enjoyyin on 2019/10/31. - */ -public abstract class AbstractReadNodeParser implements ContextNodeParser { - - @Override - public String[] getShareNodeIds(SchedulerNode node) { - //需根据节点的参数进行解析生成 - Map jobParams = node.getDssNode().getParams(); - if(jobParams == null) return null; - Map configuration =(Map) jobParams.get("configuration"); - Map runtime = (Map) configuration.get("runtime"); - if(runtime.get("category").toString().equals("node")) { - String content = runtime.get("content").toString().replace("[","").replace("]",""); - if (content != null) { - String[] res = Arrays.stream(content.split(",")).map(String::trim).toArray(String[]::new); - return res; - } else { - return null; - } - }else { - return null; - } - } - - @Override - public void setReadNodeContext(SchedulerNode node) { - // TODO: 2019/10/31 - } - - @Override - public ReadNode parseNode(SchedulerNode node) { - ReadNode readNode = createReadNode(); - readNode.setDssNode(node.getDssNode()); - readNode.setSchedulerNode(node); - readNode.setShareNodeIds(getShareNodeIds(node)); - return readNode; - } - - @Override - public SchedulerNode parseNode(DSSNode dssNode) { - SchedulerNode schedulerNode = createSchedulerNode(); - schedulerNode.setDssNode(dssNode); - return parseNode(schedulerNode); - } - - - protected abstract ReadNode createReadNode(); - - protected abstract SchedulerNode createSchedulerNode(); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/ContextNodeParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/ContextNodeParser.java deleted file mode 100644 index e8ba23db7..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/ContextNodeParser.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface ContextNodeParser extends NodeParser { - String[] getShareNodeIds(SchedulerNode node); - void setReadNodeContext(SchedulerNode node); - ReadNode parseNode(SchedulerNode node); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/FlowParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/FlowParser.java deleted file mode 100644 index 85369debc..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/FlowParser.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSJSONFlow; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface FlowParser { - - SchedulerFlow parseFlow(DSSJSONFlow flow); - - void setNodeParsers(NodeParser[] nodeParsers); - - NodeParser[] getNodeParsers(); - - Boolean ifFlowCanParse(DSSJSONFlow flow); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/NodeParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/NodeParser.java deleted file mode 100644 index ebba917a1..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/NodeParser.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.order.Order; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface NodeParser extends Order { - SchedulerNode parseNode(DSSNode dssNode); - - Boolean ifNodeCanParse(DSSNode dssNode); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/ProjectParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/ProjectParser.java deleted file mode 100644 index db00db311..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/ProjectParser.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; - -/** - * Created by enjoyyin on 2019/9/16. - */ -public interface ProjectParser { - SchedulerProject parseProject(DSSProject dssProject); - void setFlowParsers(FlowParser[] flowParser); - FlowParser[] getFlowParsers(); - -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/SendEmailNodeParser.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/SendEmailNodeParser.java deleted file mode 100644 index 1bd3606f0..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/parser/SendEmailNodeParser.java +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.parser; - -import com.webank.wedatasphere.dss.appjoint.scheduler.constant.SchedulerAppJointConstant; -import com.webank.wedatasphere.dss.common.entity.node.DSSNode; - -import java.util.Map; - -/** - * Created by enjoyyin on 2019/10/31. - */ -public abstract class SendEmailNodeParser extends AbstractReadNodeParser { - - @Override - public Boolean ifNodeCanParse(DSSNode dssNode) { - //判断是sendemail 并且category是node - Map params = dssNode.getParams(); - if(params != null && !params.isEmpty()){ - Object configuration = params.get(SchedulerAppJointConstant.CONFIGURATION); - if(configuration instanceof Map){ - Object runtime = ((Map) configuration).get(SchedulerAppJointConstant.RUNTIME); - if(runtime instanceof Map){ - Object category = ((Map) runtime).get(SchedulerAppJointConstant.CATEGORY); - if(category != null && SchedulerAppJointConstant.NODE.equals(category.toString())){ - return SchedulerAppJointConstant.SENDEMAIL_NODE_TYPE.equals(dssNode.getNodeType()); - } - } - } - } - return false; - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/service/SchedulerProjectService.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/service/SchedulerProjectService.java deleted file mode 100644 index 2bc4cea3e..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/service/SchedulerProjectService.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.service; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.project.Project; - -/** - * Created by enjoyyin on 2019/10/10. - */ -public interface SchedulerProjectService extends ProjectService { - - void publishProject(Project project, Session session) throws AppJointErrorException; -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/service/SchedulerSecurityService.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/service/SchedulerSecurityService.java deleted file mode 100644 index a9652e52c..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/service/SchedulerSecurityService.java +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.service; - -import com.webank.wedatasphere.dss.appjoint.service.SecurityService; - -/** - * Created by enjoyyin on 2019/10/12. - */ -public interface SchedulerSecurityService extends SecurityService { - void reloadToken(); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractFlowTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractFlowTuning.java deleted file mode 100644 index 42423a69d..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractFlowTuning.java +++ /dev/null @@ -1,120 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerEdge; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.constant.SchedulerAppJointConstant; -import org.apache.commons.lang.StringUtils; - -import java.util.ArrayList; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractFlowTuning implements FlowTuning { - - private NodeTuning[] nodeTunings; - - @Override - public void setNodeTunings(NodeTuning[] nodeTunings) { - this.nodeTunings = nodeTunings; - } - - @Override - public NodeTuning[] getNodeTunings() { - return this.nodeTunings; - } - - @Override - public SchedulerFlow tuningSchedulerFlow(SchedulerFlow schedulerFlow) { - //1.设置依赖，对于多个flowTuning，已经设置的就跳过 - schedulerFlow.getSchedulerNodes().forEach(node ->setDependencies(node,schedulerFlow.getSchedulerNodes(),schedulerFlow.getSchedulerEdges())); - //2.设置usreproxy - setProxyUser(schedulerFlow); - // 2.调用各种nodeTuning进行 - List schedulerNodes = schedulerFlow.getSchedulerNodes(); - for (NodeTuning nodeTuning : getNodeTunings()) { - schedulerNodes = targetNodeTuning(nodeTuning, schedulerNodes); - } - schedulerFlow.setSchedulerNodes(schedulerNodes); - //3.对子flow递归调用本方法 - List children = schedulerFlow.getChildren(); - if (children != null) { - List collect = children.stream().map(this::tuningSchedulerFlow).collect(Collectors.toList()); - schedulerFlow.setChildren(collect); - } - return schedulerFlow; - } - - private void setDependencies(SchedulerNode node,List schedulerNodes,List flowEdges){ - //设置过的flow不在进行设置和解析 - if(node.getDependencys()!= null && !node.getDependencys().isEmpty()) return; - List dependencies = resolveDependencys(node,schedulerNodes ,flowEdges); - dependencies.forEach(node::addDependency); - } - - private List resolveDependencys(SchedulerNode node,List schedulerNodes, List flowEdges) { - List dependencys = new ArrayList<>(); - flowEdges.forEach(edge -> { - if (edge.getDssEdge().getTarget().equals(node.getId())) { - dependencys.add(schedulerNodes.stream().filter(n ->edge.getDssEdge().getSource().equals(n.getId())).findFirst().get().getName()); - } - }); - - return dependencys; - } - - private List targetNodeTuning(NodeTuning nodeTuning, List schedulerNodes) { - return schedulerNodes.stream().map(node -> invokeNodeTuning(nodeTuning, node)).collect(Collectors.toList()); - } - - private SchedulerNode invokeNodeTuning(NodeTuning nodeTuning, SchedulerNode schedulerNode) { - if (nodeTuning.ifNodeCanTuning(schedulerNode)) { - return nodeTuning.tuningSchedulerNode(schedulerNode); - } else { - return schedulerNode; - } - } - - private String getProxyUser(SchedulerFlow schedulerFlow) { - if(schedulerFlow.getFlowProperties() == null) { - return null; - } - StringBuilder sb = new StringBuilder(); - schedulerFlow.getFlowProperties().forEach( map -> { - Object value = map.get(SchedulerAppJointConstant.PROXY_USER); - if(value != null && StringUtils.isNotBlank(value.toString())) { - sb.append(value.toString()); - } - }); - return sb.toString(); - } - - private void setProxyUser(SchedulerFlow schedulerFlow) { - String proxyUser = getProxyUser(schedulerFlow); - if(StringUtils.isNotBlank(proxyUser)) { - schedulerFlow.getSchedulerNodes().forEach(node -> node.getDssNode().setUserProxy(proxyUser)); - schedulerFlow.setUserProxy(proxyUser); - } - } - -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractNodeTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractNodeTuning.java deleted file mode 100644 index 9d15de9bd..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractNodeTuning.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractNodeTuning implements NodeTuning { - @Override - public SchedulerNode tuningSchedulerNode(SchedulerNode node) { - return node; - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractProjectTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractProjectTuning.java deleted file mode 100644 index 913ad636f..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractProjectTuning.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.AbstractSchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; - -import java.util.Arrays; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Created by enjoyyin on 2019/9/25. - */ -public abstract class AbstractProjectTuning implements ProjectTuning { - - private FlowTuning [] flowTunings; - - @Override - public void setFlowTunings(FlowTuning[] flowTunings) { - this.flowTunings = flowTunings; - } - - @Override - public FlowTuning[] getFlowTunings() { - return this.flowTunings; - } - - @Override - public SchedulerProject tuningSchedulerProject(SchedulerProject schedulerProject) { - // TODO: 2019/9/25 这里强转是因为没有把 schedulerProject由接口修改为普通类 - // TODO: 2019/9/25 这里如果flow - AbstractSchedulerProject abstractSchedulerProject = (AbstractSchedulerProject) schedulerProject; - List schedulerFlows = abstractSchedulerProject.getSchedulerFlows(); - List flowTunings = Arrays.stream(getFlowTunings()).sorted((t1, t2) -> t2.getOrder() - t1.getOrder()).collect(Collectors.toList()); - for (FlowTuning flowTuning : flowTunings) { - schedulerFlows = targetFlowTuning(flowTuning,schedulerFlows); - } - abstractSchedulerProject.setSchedulerFlows(schedulerFlows); - return abstractSchedulerProject; - } - - private List targetFlowTuning(FlowTuning flowTuning,List schedulerFlows){ - return schedulerFlows.stream().map(flow ->invokeFlowTuning(flowTuning,flow)).collect(Collectors.toList()); - } - - private SchedulerFlow invokeFlowTuning(FlowTuning flowTuning,SchedulerFlow schedulerFlow){ - if (flowTuning.ifFlowCanTuning(schedulerFlow)){ - return flowTuning.tuningSchedulerFlow(schedulerFlow); - }else { - return schedulerFlow; - } - } -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractShareNodeFlowTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractShareNodeFlowTuning.java deleted file mode 100644 index 81dc0d5d3..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/AbstractShareNodeFlowTuning.java +++ /dev/null @@ -1,96 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; - -import java.util.*; -import java.util.function.Function; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -/** - * Created by enjoyyin on 2019/10/31. - */ -public abstract class AbstractShareNodeFlowTuning extends AbstractFlowTuning implements ContextShareableFlowTuning { - - @Override - public SchedulerFlow tuningSchedulerFlow(SchedulerFlow schedulerFlow) { - List schedulerNodes = schedulerFlow.getSchedulerNodes(); - List tuningSchedulerNode = new ArrayList<>(); - List readNodeList = getReadNodes(schedulerFlow); - ReadNode[] schedulerNodes1 = new ReadNode[readNodeList.size()]; - Map shareNodeMap = getShareNodes(schedulerFlow,readNodeList.toArray(schedulerNodes1)); - schedulerNodes.stream().forEach(x ->{ - Map.Entry findShareNode = shareNodeMap.entrySet().stream().filter(y -> - (y.getKey().getName()).equals(x.getName()) - ).findFirst().orElse(null); - if(findShareNode != null ){ - tuningSchedulerNode.add(findShareNode.getKey()); - - }else { - tuningSchedulerNode.add(x); - } - }); - schedulerFlow.setSchedulerNodes(tuningSchedulerNode); - return super.tuningSchedulerFlow(schedulerFlow); - } - - @Override - public List getReadNodes(SchedulerFlow flow) { - List res = flow.getSchedulerNodes().stream().filter(x -> x instanceof ReadNode).collect(Collectors.toList()); - return res; - } - - @Override - public Map getShareNodes(SchedulerFlow flow, ReadNode[] readNodes) { - List list = new ArrayList<>(); - Map res = new HashMap<>(); - //遍历readNodes，将NodeIds转为name的集合,过滤掉删除了节点但是还滞留在content里面的id - Arrays.stream(readNodes).filter(rn ->rn.getShareNodeIds() != null).forEach(rn ->{ - rn.setShareNodeIds(Arrays.stream(rn.getShareNodeIds()).filter(id -> flow.getSchedulerNodes().stream().anyMatch(sn -> id.equals(sn.getId()))). - map(id -> flow.getSchedulerNodes().stream().filter(sn -> id.equals(sn.getId())).findFirst().get().getName()).toArray(String[]::new)); - }); - Stream.of(readNodes).forEach(x -> - { - if(x.getShareNodeIds() != null) { - list.addAll(Arrays.asList(x.getShareNodeIds())); - } - } - ); - Map nameAndNumMap = list.stream().collect(Collectors.groupingBy(Function.identity(),Collectors.counting())); - nameAndNumMap.keySet().forEach(key ->{ - - SchedulerNode schedulerNode = flow.getSchedulerNodes().stream().filter(x->x.getName().equals(key.toString())).findFirst().orElse(null); - if(schedulerNode != null) { - int shareTimes = (nameAndNumMap.get(key)).intValue(); - ShareNode shareNode = createShareNode(); - shareNode.setDssNode(schedulerNode.getDssNode()); - shareNode.setSchedulerNode(schedulerNode); - shareNode.setShareTimes(shareTimes); - res.put(shareNode, shareTimes); - } - }); - return res; - } - - protected abstract ShareNode createShareNode(); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ContextNodeTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ContextNodeTuning.java deleted file mode 100644 index c4cea46b4..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ContextNodeTuning.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface ContextNodeTuning { - void setShareNodeContext(ShareNode node); - ShareNode tuningSchedulerNode(ShareNode node); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ContextShareableFlowTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ContextShareableFlowTuning.java deleted file mode 100644 index 0c4706f88..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ContextShareableFlowTuning.java +++ /dev/null @@ -1,34 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ShareNode; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.ReadNode; - -import java.util.List; -import java.util.Map; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface ContextShareableFlowTuning { - List getReadNodes(SchedulerFlow flow); - Map getShareNodes(SchedulerFlow flow, ReadNode[] readNodes); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/FlowTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/FlowTuning.java deleted file mode 100644 index 2bc523330..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/FlowTuning.java +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerFlow; -import com.webank.wedatasphere.dss.appjoint.scheduler.order.Order; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface FlowTuning extends Order { - SchedulerFlow tuningSchedulerFlow(SchedulerFlow schedulerFlow); - void setNodeTunings(NodeTuning[] NodeTunings); - NodeTuning[] getNodeTunings(); - Boolean ifFlowCanTuning(SchedulerFlow schedulerFlow); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/NodeTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/NodeTuning.java deleted file mode 100644 index dedd846d3..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/NodeTuning.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerNode; - -/** - * Created by enjoyyin on 2019/9/7. - */ -public interface NodeTuning { - SchedulerNode tuningSchedulerNode(SchedulerNode node); - Boolean ifNodeCanTuning(SchedulerNode node); -} diff --git a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ProjectTuning.java b/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ProjectTuning.java deleted file mode 100644 index b2b40516b..000000000 --- a/dss-scheduler-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/scheduler/tuning/ProjectTuning.java +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.scheduler.tuning; - -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; - -/** - * Created by enjoyyin on 2019/9/18 - */ -public interface ProjectTuning { - SchedulerProject tuningSchedulerProject(SchedulerProject schedulerProject); - void setFlowTunings(FlowTuning[] flowTunings); - FlowTuning[] getFlowTunings(); -} diff --git a/dss-server/bin/start-dss-server.sh b/dss-server/bin/start-dss-server.sh deleted file mode 100644 index 518cd8da1..000000000 --- a/dss-server/bin/start-dss-server.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/bash -cd `dirname $0` -cd .. -HOME=`pwd` - -export SERVER_PID=$HOME/bin/linkis.pid -export SERVER_LOG_PATH=$HOME/logs -export SERVER_CLASS=com.webank.wedatasphere.dss.DSSSpringApplication - -if test -z "$SERVER_HEAP_SIZE" -then - export SERVER_HEAP_SIZE="512M" -fi - -if test -z "$SERVER_JAVA_OPTS" -then - export SERVER_JAVA_OPTS=" -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$HOME/logs/linkis-gc.log" -fi - -if [[ -f "${SERVER_PID}" ]]; then - pid=$(cat ${SERVER_PID}) - if kill -0 ${pid} >/dev/null 2>&1; then - echo "Server is already running." - exit 1 - fi -fi - -nohup java $SERVER_JAVA_OPTS -cp $HOME/conf:$HOME/lib/* $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/linkis.out & -pid=$! -if [[ -z "${pid}" ]]; then - echo "server $SERVER_NAME start failed!" - exit 1 -else - echo "server $SERVER_NAME start succeeded!" - echo $pid > $SERVER_PID - sleep 1 -fi - - diff --git a/dss-server/bin/stop-dss-server.sh b/dss-server/bin/stop-dss-server.sh deleted file mode 100644 index f8b94c276..000000000 --- a/dss-server/bin/stop-dss-server.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -cd `dirname $0` -cd .. -HOME=`pwd` - -export FLOW_ENTRANCE_HOME_PID=$HOME/bin/linkis.pid - -function wait_for_FLOW_ENTRANCE_HOME_to_die() { - local pid - local count - pid=$1 - timeout=$2 - count=0 - timeoutTime=$(date "+%s") - let "timeoutTime+=$timeout" - currentTime=$(date "+%s") - forceKill=1 - - while [[ $currentTime -lt $timeoutTime ]]; do - $(kill ${pid} > /dev/null 2> /dev/null) - if kill -0 ${pid} > /dev/null 2>&1; then - sleep 3 - else - forceKill=0 - break - fi - currentTime=$(date "+%s") - done - - if [[ forceKill -ne 0 ]]; then - $(kill -9 ${pid} > /dev/null 2> /dev/null) - fi -} - -if [[ ! -f "${FLOW_ENTRANCE_HOME_PID}" ]]; then - echo "DSS SERVER is not running" -else - pid=$(cat ${FLOW_ENTRANCE_HOME_PID}) - if [[ -z "${pid}" ]]; then - echo "DSS SERVER is not running" - else - wait_for_FLOW_ENTRANCE_HOME_to_die $pid 40 - $(rm -f ${FLOW_ENTRANCE_HOME_PID}) - echo "DSS SERVER is stopped." - fi -fi diff --git a/dss-server/pom.xml b/dss-server/pom.xml deleted file mode 100644 index 76d1446dd..000000000 --- a/dss-server/pom.xml +++ /dev/null @@ -1,237 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-server - - - - - org.mybatis.spring.boot - mybatis-spring-boot-starter - 1.3.2 - - - spring-boot-starter - org.springframework.boot - - - spring-boot-autoconfigure - org.springframework.boot - - - spring-beans - org.springframework - - - spring-core - org.springframework - - - - - - com.webank.wedatasphere.dss - dss-appjoint-loader - ${dss.version} - - - - - - - - - - - - - - - - - - - - org.springframework.cloud - spring-cloud-starter-netflix-eureka-client - 2.0.0.RELEASE - - - logback-classic - ch.qos.logback - - - log4j-to-slf4j - org.apache.logging.log4j - - - gson - com.google.code.gson - - - jsr311-api - javax.ws.rs - - - - - - org.springframework.cloud - spring-cloud-starter-openfeign - 2.0.0.RELEASE - - - - com.webank.wedatasphere.dss - dss-application - ${dss.version} - - - - linkis-bmlclient - - - gson - com.google.code.gson - - - com.webank.wedatasphere.linkis - ${linkis.version} - - - - com.webank.wedatasphere.linkis - linkis-cloudRPC - ${linkis.version} - - - spring-cloud-starter-netflix-eureka-client - org.springframework.cloud - - - spring-cloud-starter-openfeign - org.springframework.cloud - - - - - - junit - junit - 4.12 - test - - - com.webank.wedatasphere.dss - dss-scheduler-appjoint-core - 0.9.1 - - - com.webank.wedatasphere.dss - dss-user-manager - 0.9.1 - compile - - - com.github.rholder - guava-retrying - 2.0.0 - - - dom4j - dom4j - 1.6.1 - - - com.typesafe - config - 1.4.1 - - - xml-apis - xml-apis - 1.4.01 - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - dss-server - false - false - - src/main/assembly/distribution.xml - - - - - - - src/main/java - - **/*.xml - - - - - - \ No newline at end of file diff --git a/dss-server/src/main/assembly/distribution.xml b/dss-server/src/main/assembly/distribution.xml deleted file mode 100644 index 0da5c182c..000000000 --- a/dss-server/src/main/assembly/distribution.xml +++ /dev/null @@ -1,340 +0,0 @@ - - - - dss-server - - zip - - true - dss-server - - - - - - lib - true - true - false - true - true - - antlr:antlr:jar - aopalliance:aopalliance:jar - asm:asm:jar - cglib:cglib:jar - - - - - - - com.fasterxml.jackson.core:jackson-annotations:jar - com.fasterxml.jackson.core:jackson-core:jar - com.fasterxml.jackson.core:jackson-databind:jar - - com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar - com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar - com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar - com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar - com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar - com.fasterxml.jackson.module:jackson-module-parameter-names:jar - com.fasterxml.jackson.module:jackson-module-paranamer:jar - com.fasterxml.jackson.module:jackson-module-scala_2.11:jar - - - com.google.code.findbugs:annotations:jar - - com.google.code.gson:gson:jar - com.google.guava:guava:jar - com.google.inject:guice:jar - com.google.protobuf:protobuf-java:jar - com.netflix.archaius:archaius-core:jar - com.netflix.eureka:eureka-client:jar - com.netflix.eureka:eureka-core:jar - com.netflix.hystrix:hystrix-core:jar - com.netflix.netflix-commons:netflix-commons-util:jar - com.netflix.netflix-commons:netflix-eventbus:jar - com.netflix.netflix-commons:netflix-infix:jar - com.netflix.netflix-commons:netflix-statistics:jar - com.netflix.ribbon:ribbon:jar - com.netflix.ribbon:ribbon-core:jar - com.netflix.ribbon:ribbon-eureka:jar - com.netflix.ribbon:ribbon-httpclient:jar - com.netflix.ribbon:ribbon-loadbalancer:jar - com.netflix.ribbon:ribbon-transport:jar - com.netflix.servo:servo-core:jar - com.sun.jersey.contribs:jersey-apache-client4:jar - com.sun.jersey:jersey-client:jar - com.sun.jersey:jersey-core:jar - com.sun.jersey:jersey-json:jar - com.sun.jersey:jersey-server:jar - com.sun.jersey:jersey-servlet:jar - com.sun.xml.bind:jaxb-impl:jar - com.thoughtworks.paranamer:paranamer:jar - com.thoughtworks.xstream:xstream:jar - com.webank.wedatasphere.linkis:linkis-common:jar - com.webank.wedatasphere.linkis:linkis-module:jar - commons-beanutils:commons-beanutils:jar - - commons-cli:commons-cli:jar - commons-codec:commons-codec:jar - commons-collections:commons-collections:jar - commons-configuration:commons-configuration:jar - commons-daemon:commons-daemon:jar - commons-dbcp:commons-dbcp:jar - - commons-httpclient:commons-httpclient:jar - commons-io:commons-io:jar - commons-jxpath:commons-jxpath:jar - commons-lang:commons-lang:jar - commons-logging:commons-logging:jar - commons-net:commons-net:jar - commons-pool:commons-pool:jar - io.micrometer:micrometer-core:jar - io.netty:netty:jar - io.netty:netty-all:jar - io.netty:netty-buffer:jar - io.netty:netty-codec:jar - io.netty:netty-codec-http:jar - io.netty:netty-common:jar - io.netty:netty-handler:jar - io.netty:netty-transport:jar - io.netty:netty-transport-native-epoll:jar - io.reactivex:rxjava:jar - io.reactivex:rxnetty:jar - io.reactivex:rxnetty-contexts:jar - io.reactivex:rxnetty-servo:jar - javax.activation:activation:jar - javax.annotation:javax.annotation-api:jar - javax.inject:javax.inject:jar - javax.servlet:javax.servlet-api:jar - javax.servlet.jsp:jsp-api:jar - javax.validation:validation-api:jar - javax.websocket:javax.websocket-api:jar - javax.ws.rs:javax.ws.rs-api:jar - javax.xml.bind:jaxb-api:jar - javax.xml.stream:stax-api:jar - joda-time:joda-time:jar - log4j:log4j:jar - mysql:mysql-connector-java:jar - org.antlr:antlr-runtime:jar - org.antlr:stringtemplate:jar - org.apache.commons:commons-compress:jar - org.apache.commons:commons-math:jar - org.apache.commons:commons-math3:jar - org.apache.curator:curator-client:jar - org.apache.curator:curator-framework:jar - org.apache.curator:curator-recipes:jar - org.apache.directory.api:api-asn1-api:jar - org.apache.directory.api:api-util:jar - org.apache.directory.server:apacheds-i18n:jar - org.apache.directory.server:apacheds-kerberos-codec:jar - org.apache.hadoop:hadoop-annotations:jar - org.apache.hadoop:hadoop-auth:jar - org.apache.hadoop:hadoop-common:jar - org.apache.hadoop:hadoop-hdfs:jar - org.apache.htrace:htrace-core:jar - org.apache.httpcomponents:httpclient:jar - org.apache.httpcomponents:httpcore:jar - org.apache.logging.log4j:log4j-api:jar - org.apache.logging.log4j:log4j-core:jar - org.apache.logging.log4j:log4j-jul:jar - org.apache.logging.log4j:log4j-slf4j-impl:jar - org.apache.zookeeper:zookeeper:jar - org.aspectj:aspectjweaver:jar - org.bouncycastle:bcpkix-jdk15on:jar - org.bouncycastle:bcprov-jdk15on:jar - org.codehaus.jackson:jackson-jaxrs:jar - org.codehaus.jackson:jackson-xc:jar - org.codehaus.jettison:jettison:jar - org.codehaus.woodstox:stax2-api:jar - org.codehaus.woodstox:woodstox-core-asl:jar - org.eclipse.jetty:jetty-annotations:jar - org.eclipse.jetty:jetty-client:jar - org.eclipse.jetty:jetty-continuation:jar - org.eclipse.jetty:jetty-http:jar - org.eclipse.jetty:jetty-io:jar - - org.eclipse.jetty:jetty-plus:jar - org.eclipse.jetty:jetty-security:jar - org.eclipse.jetty:jetty-server:jar - org.eclipse.jetty:jetty-servlet:jar - org.eclipse.jetty:jetty-servlets:jar - org.eclipse.jetty:jetty-util:jar - org.eclipse.jetty:jetty-webapp:jar - org.eclipse.jetty:jetty-xml:jar - org.eclipse.jetty.websocket:javax-websocket-client-impl:jar - org.eclipse.jetty.websocket:javax-websocket-server-impl:jar - org.eclipse.jetty.websocket:websocket-api:jar - org.eclipse.jetty.websocket:websocket-client:jar - org.eclipse.jetty.websocket:websocket-common:jar - org.eclipse.jetty.websocket:websocket-server:jar - org.eclipse.jetty.websocket:websocket-servlet:jar - org.fusesource.leveldbjni:leveldbjni-all:jar - org.glassfish.hk2:class-model:jar - org.glassfish.hk2:config-types:jar - org.glassfish.hk2.external:aopalliance-repackaged:jar - org.glassfish.hk2.external:asm-all-repackaged:jar - org.glassfish.hk2.external:bean-validator:jar - org.glassfish.hk2.external:javax.inject:jar - org.glassfish.hk2:hk2:jar - org.glassfish.hk2:hk2-api:jar - org.glassfish.hk2:hk2-config:jar - org.glassfish.hk2:hk2-core:jar - org.glassfish.hk2:hk2-locator:jar - org.glassfish.hk2:hk2-runlevel:jar - org.glassfish.hk2:hk2-utils:jar - org.glassfish.hk2:osgi-resource-locator:jar - org.glassfish.hk2:spring-bridge:jar - org.glassfish.jersey.bundles:jaxrs-ri:jar - org.glassfish.jersey.bundles.repackaged:jersey-guava:jar - org.glassfish.jersey.containers:jersey-container-servlet:jar - org.glassfish.jersey.containers:jersey-container-servlet-core:jar - org.glassfish.jersey.core:jersey-client:jar - org.glassfish.jersey.core:jersey-common:jar - org.glassfish.jersey.core:jersey-server:jar - org.glassfish.jersey.ext:jersey-entity-filtering:jar - org.glassfish.jersey.ext:jersey-spring3:jar - org.glassfish.jersey.media:jersey-media-jaxb:jar - org.glassfish.jersey.media:jersey-media-json-jackson:jar - org.glassfish.jersey.media:jersey-media-multipart:jar - org.hdrhistogram:HdrHistogram:jar - org.javassist:javassist:jar - org.json4s:json4s-ast_2.11:jar - org.json4s:json4s-core_2.11:jar - org.json4s:json4s-jackson_2.11:jar - - org.jvnet.mimepull:mimepull:jar - org.jvnet:tiger-types:jar - org.latencyutils:LatencyUtils:jar - org.mortbay.jasper:apache-el:jar - org.mortbay.jetty:jetty:jar - org.mortbay.jetty:jetty-util:jar - org.ow2.asm:asm-analysis:jar - org.ow2.asm:asm-commons:jar - org.ow2.asm:asm-tree:jar - org.reflections:reflections:jar - org.scala-lang.modules:scala-parser-combinators_2.11:jar - org.scala-lang.modules:scala-xml_2.11:jar - org.scala-lang:scala-compiler:jar - org.scala-lang:scala-library:jar - org.scala-lang:scala-reflect:jar - org.scala-lang:scalap:jar - org.slf4j:jul-to-slf4j:jar - org.slf4j:slf4j-api:jar - org.springframework.boot:spring-boot:jar - org.springframework.boot:spring-boot-actuator:jar - org.springframework.boot:spring-boot-actuator-autoconfigure:jar - org.springframework.boot:spring-boot-autoconfigure:jar - org.springframework.boot:spring-boot-starter:jar - org.springframework.boot:spring-boot-starter-actuator:jar - org.springframework.boot:spring-boot-starter-aop:jar - org.springframework.boot:spring-boot-starter-jetty:jar - org.springframework.boot:spring-boot-starter-json:jar - org.springframework.boot:spring-boot-starter-log4j2:jar - org.springframework.boot:spring-boot-starter-web:jar - org.springframework.cloud:spring-cloud-commons:jar - org.springframework.cloud:spring-cloud-config-client:jar - org.springframework.cloud:spring-cloud-context:jar - - org.springframework.cloud:spring-cloud-netflix-core:jar - org.springframework.cloud:spring-cloud-netflix-eureka-client:jar - - org.springframework.cloud:spring-cloud-starter:jar - org.springframework.cloud:spring-cloud-starter-config:jar - org.springframework.cloud:spring-cloud-starter-eureka:jar - org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar - org.springframework.cloud:spring-cloud-starter-netflix-eureka-client:jar - org.springframework.cloud:spring-cloud-starter-netflix-ribbon:jar - org.springframework.security:spring-security-crypto:jar - org.springframework.security:spring-security-rsa:jar - org.springframework:spring-aop:jar - org.springframework:spring-beans:jar - org.springframework:spring-context:jar - org.springframework:spring-core:jar - org.springframework:spring-expression:jar - org.springframework:spring-jcl:jar - org.springframework:spring-web:jar - org.springframework:spring-webmvc:jar - org.tukaani:xz:jar - org.yaml:snakeyaml:jar - - xerces:xercesImpl:jar - xmlenc:xmlenc:jar - xmlpull:xmlpull:jar - xpp3:xpp3_min:jar - - - - - - - ${basedir}/src/main/resources - - * - - 0777 - conf - unix - - - ${project.parent.basedir}/dss-user-manager/src/main/resources/config - - * - - 0777 - conf/config - unix - - - ${project.parent.basedir}/dss-user-manager/src/main/resources/default - - * - - 0777 - conf/default - unix - - - ${basedir}/bin - - * - - 0777 - bin - unix - - - . - - */** - - logs - - - - - diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/DSSSpringApplication.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/DSSSpringApplication.java deleted file mode 100644 index 8a7344f9b..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/DSSSpringApplication.java +++ /dev/null @@ -1,222 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss; - -import com.webank.wedatasphere.linkis.DataWorkCloudApplication; -import com.webank.wedatasphere.linkis.common.ServiceInstance; -import com.webank.wedatasphere.linkis.common.conf.BDPConfiguration; -import com.webank.wedatasphere.linkis.common.conf.Configuration; -import com.webank.wedatasphere.linkis.common.exception.DWCException; -import com.webank.wedatasphere.linkis.common.utils.Utils; -import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; -import com.webank.wedatasphere.linkis.server.conf.DataWorkCloudCustomExcludeFilter; -import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration; -import org.apache.commons.lang.StringUtils; -import org.apache.commons.logging.Log; -import org.apache.commons.logging.LogFactory; -import org.eclipse.jetty.server.Handler; -import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.webapp.WebAppContext; -import org.springframework.boot.SpringApplication; -import org.springframework.boot.autoconfigure.SpringBootApplication; -import org.springframework.boot.builder.SpringApplicationBuilder; -import org.springframework.boot.context.event.ApplicationPreparedEvent; -import org.springframework.boot.web.embedded.jetty.JettyServerCustomizer; -import org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory; -import org.springframework.boot.web.server.WebServerFactoryCustomizer; -import org.springframework.boot.web.servlet.support.SpringBootServletInitializer; -import org.springframework.cloud.client.discovery.EnableDiscoveryClient; -import org.springframework.cloud.context.config.annotation.RefreshScope; -import org.springframework.cloud.context.scope.refresh.RefreshScopeRefreshedEvent; -import org.springframework.context.ApplicationContext; -import org.springframework.context.ApplicationListener; -import org.springframework.context.ConfigurableApplicationContext; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.ComponentScan; -import org.springframework.context.annotation.FilterType; -import org.springframework.core.env.CompositePropertySource; -import org.springframework.core.env.Environment; -import org.springframework.core.env.PropertySource; -import org.springframework.core.env.StandardEnvironment; -import org.springframework.web.filter.CharacterEncodingFilter; - -import javax.servlet.DispatcherType; -import javax.ws.rs.ext.RuntimeDelegate; -import java.lang.reflect.Field; -import java.util.EnumSet; - -/** - * Created by enjoyyin on 2018/8/6. - */ -@SpringBootApplication -@EnableDiscoveryClient -@RefreshScope -@ComponentScan(basePackages = "com.webank.wedatasphere", - excludeFilters = @ComponentScan.Filter(type = FilterType.CUSTOM, classes = {DataWorkCloudCustomExcludeFilter.class})) -public class DSSSpringApplication extends SpringBootServletInitializer { - private static final Log logger = LogFactory.getLog(DSSSpringApplication.class); - - private static ConfigurableApplicationContext applicationContext; - private static ServiceInstance serviceInstance; - - public static ApplicationContext getApplicationContext() { - return applicationContext; - } - - public static void main(String[] args) throws Exception { - RuntimeDelegate.setInstance(new org.glassfish.jersey.internal.RuntimeDelegateImpl()); - final SpringApplication application = new SpringApplication(DSSSpringApplication.class); - application.addListeners(new ApplicationListener(){ - public void onApplicationEvent(ApplicationPreparedEvent applicationPreparedEvent) { - logger.info("add config from config server..."); - if(applicationContext == null) { - applicationContext = applicationPreparedEvent.getApplicationContext(); - try { - setApplicationContext(applicationContext); - } catch (Exception e) { - logger.error(e); - } - } - addRemoteConfig(); - logger.info("initialize DataWorkCloud spring application..."); - initDWCApplication(); - } - }); - application.addListeners(new ApplicationListener() { - public void onApplicationEvent(RefreshScopeRefreshedEvent applicationEvent) { - logger.info("refresh config from config server..."); - updateRemoteConfig(); - } - }); - String listeners = ServerConfiguration.BDP_SERVER_SPRING_APPLICATION_LISTENERS().getValue(); - if(StringUtils.isNotBlank(listeners)) { - for (String listener : listeners.split(",")) { - application.addListeners((ApplicationListener) Class.forName(listener).newInstance()); - } - } - applicationContext = application.run(args); - setApplicationContext(applicationContext); - } - - private static void setApplicationContext(ConfigurableApplicationContext applicationContext) throws Exception { - Field applicationContextField = DataWorkCloudApplication.class.getDeclaredField("applicationContext"); - applicationContextField.setAccessible(true); - applicationContextField.set(null,applicationContext); - } - - public static void updateRemoteConfig() { - addOrUpdateRemoteConfig(applicationContext.getEnvironment(), true); - } - - public static void addRemoteConfig() { - addOrUpdateRemoteConfig(applicationContext.getEnvironment(), false); - } - - private static void addOrUpdateRemoteConfig(Environment env, boolean isUpdateOrNot) { - StandardEnvironment environment = (StandardEnvironment) env; - PropertySource propertySource = environment.getPropertySources().get("bootstrapProperties"); - if(propertySource == null) { - return; - } - CompositePropertySource source = (CompositePropertySource) propertySource; - for (String key: source.getPropertyNames()) { - Object val = source.getProperty(key); - if(val == null) { - continue; - } - if(isUpdateOrNot) { - logger.info("update remote config => " + key + " = " + source.getProperty(key)); - BDPConfiguration.set(key, val.toString()); - } else { - logger.info("add remote config => " + key + " = " + source.getProperty(key)); - BDPConfiguration.setIfNotExists(key, val.toString()); - } - } - } - - private static void initDWCApplication() { - serviceInstance = new ServiceInstance(); - serviceInstance.setApplicationName(applicationContext.getEnvironment().getProperty("spring.application.name")); - serviceInstance.setInstance(Utils.getComputerName() + ":" + applicationContext.getEnvironment().getProperty("server.port")); - try { - setServiceInstance(serviceInstance); - } catch (Exception e) { - logger.error(e); - } - DWCException.setApplicationName(serviceInstance.getApplicationName()); - DWCException.setHostname(Utils.getComputerName()); - DWCException.setHostPort(Integer.parseInt(applicationContext.getEnvironment().getProperty("server.port","9004"))); - } - - private static void setServiceInstance(ServiceInstance serviceInstance) throws Exception { - Field applicationContextField = DataWorkCloudApplication.class.getDeclaredField("serviceInstance"); - applicationContextField.setAccessible(true); - applicationContextField.set(null,serviceInstance); - } - - public static ServiceInstance getServiceInstance() { - return serviceInstance; - } - - public static String getApplicationName() { - return serviceInstance.getApplicationName(); - } - - public static String getInstance() { - return serviceInstance.getInstance(); - } - - public static void setProperty(String key, String value) { - BDPConfiguration.set(key, value); - } - - public static void setPropertyIfNotExists(String key, String value) { - BDPConfiguration.setIfNotExists(key, value); - } - - @Override - protected SpringApplicationBuilder configure(SpringApplicationBuilder builder) { - return builder.sources(DSSSpringApplication.class); - } - - @Bean - public WebServerFactoryCustomizer jettyFactoryCustomizer() { - return new WebServerFactoryCustomizer() { - public void customize(JettyServletWebServerFactory jettyServletWebServerFactory) { - jettyServletWebServerFactory.addServerCustomizers(new JettyServerCustomizer() { - public void customize(Server server) { - Handler[] childHandlersByClass = server.getChildHandlersByClass(WebAppContext.class); - final WebAppContext webApp = (WebAppContext) childHandlersByClass[0]; - FilterHolder filterHolder = new FilterHolder(CharacterEncodingFilter.class); - filterHolder.setInitParameter("encoding", Configuration.BDP_ENCODING().getValue()); - filterHolder.setInitParameter("forceEncoding", "true"); - webApp.addFilter(filterHolder, "/*", EnumSet.allOf(DispatcherType.class)); - BDPJettyServerHelper.setupRestApiContextHandler(webApp); - if(ServerConfiguration.BDP_SERVER_SOCKET_MODE().getValue()) { - BDPJettyServerHelper.setupControllerServer(webApp); - } - if(!ServerConfiguration.BDP_SERVER_DISTINCT_MODE().getValue()) { - BDPJettyServerHelper.setupWebAppContext(webApp); - } - } - }); - } - }; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/conf/DSSServerConf.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/conf/DSSServerConf.java deleted file mode 100644 index aa4a454a7..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/conf/DSSServerConf.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.conf; - - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; -import com.webank.wedatasphere.linkis.common.conf.CommonVars$; - - -public class DSSServerConf { - public static final CommonVars PUBLISH_TIMEOUT = CommonVars$.MODULE$.apply("wds.dss.server.publish.timeout",60 * 10); - public static final CommonVars CACHE_TIMEOUT = CommonVars$.MODULE$.apply("wds.dss.server.cache.timeout",1000 * 60 * 60); - - public static final CommonVars BDP_SERVER_MYBATIS_MAPPER_LOCATIONS = CommonVars.apply("wds.linkis.server.mybatis.mapperLocations", ""); - public static final CommonVars BDP_SERVER_MYBATIS_TYPEALIASESPACKAGE = CommonVars.apply("wds.linkis.server.mybatis.typeAliasesPackage", ""); - public static final CommonVars BDP_SERVER_MYBATIS_CONFIGLOCATION = CommonVars.apply("wds.linkis.server.mybatis.configLocation", "classpath:/mybatis-config.xml"); - public static final CommonVars BDP_SERVER_MYBATIS_BASEPACKAGE = CommonVars.apply("wds.linkis.server.mybatis.BasePackage", ""); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_URL = CommonVars.apply("wds.linkis.server.mybatis.datasource.url", ""); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_USERNAME = CommonVars.apply("wds.linkis.server.mybatis.datasource.username", ""); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_PASSWORD = CommonVars.apply("wds.linkis.server.mybatis.datasource.password", ""); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_DRIVER_CLASS_NAME = CommonVars.apply("wds.linkis.server.mybatis.datasource.driver-class-name", "com.mysql.jdbc.Driver"); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_INITIALSIZE = CommonVars.apply("wds.linkis.server.mybatis.datasource.initialSize", new Integer(1)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_MINIDLE = CommonVars.apply("wds.linkis.server.mybatis.datasource.minIdle", new Integer(1)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_MAXACTIVE = CommonVars.apply("wds.linkis.server.mybatis.datasource.maxActive", new Integer(20)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_MAXWAIT = CommonVars.apply("wds.linkis.server.mybatis.datasource.maxWait", new Integer(6000)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_TBERM = CommonVars.apply("wds.linkis.server.mybatis.datasource.timeBetweenEvictionRunsMillis", new Integer(60000)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_MEITM = CommonVars.apply("wds.linkis.server.mybatis.datasource.minEvictableIdleTimeMillis", new Integer(300000)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_VALIDATIONQUERY = CommonVars.apply("wds.linkis.server.mybatis.datasource.validationQuery", "SELECT 1"); - - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_TESTWHILEIDLE = CommonVars.apply("wds.linkis.server.mybatis.datasource.testWhileIdle", new Boolean(true)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_TESTONBORROW = CommonVars.apply("wds.linkis.server.mybatis.datasource.testOnBorrow", new Boolean(false)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_TESTONRETURN = CommonVars.apply("wds.linkis.server.mybatis.datasource.testOnReturn", new Boolean(false)); - public static final CommonVars BDP_SERVER_MYBATIS_DATASOURCE_POOLPREPAREDSTATEMENTS = CommonVars.apply("wds.linkis.server.mybatis.datasource.poolPreparedStatements", new Boolean(true)); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/constant/DSSServerConstant.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/constant/DSSServerConstant.java deleted file mode 100644 index 2c5711e15..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/constant/DSSServerConstant.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.constant; - - -public class DSSServerConstant { - public static final String DSS_PROJECT_FIRST_VERSION = "v000001"; - public static final String DSS_PROJECT_FIRST_VERSION_COMMENT = "first version"; - public static final String DSS_PROJECT_SOURCE = "create by user"; - public static final String DSS_WORKSPACE_SOURCE = "create by user"; - public static final String PROJECT_VERSION_ID = "projectVersionID"; - public static final String PUBLISH_FLOW_REPORT_FORMATE = "the workflow name is% s, the version number is% s, and the workflow content is empty. Please modify or delete it by yourself"; - public static final String EMVEDDEDFLOWID = "\"embeddedFlowId\":"; - public static final String VERSION_FORMAT = "%06d"; - public static final String VERSION_PREFIX = "v"; - public static final String SUPER_USER_LOGIN_ERROR = "please login with super user"; - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/DSSUserMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/DSSUserMapper.java deleted file mode 100644 index 31d4974e1..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/DSSUserMapper.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - -import com.webank.wedatasphere.dss.application.entity.DSSUser; - -public interface DSSUserMapper { - Long getUserID(String userName); - - String getuserName(Long userID); - - DSSUser getUserByName(String username); - - void registerDSSUser(DSSUser userDb); - - void updateUserFirstLogin(Long userId); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/FlowMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/FlowMapper.java deleted file mode 100644 index ea830d7cf..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/FlowMapper.java +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - - -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlowVersion; -import org.apache.ibatis.annotations.Param; -import org.springframework.dao.DuplicateKeyException; - -import java.util.List; - - -public interface FlowMapper { - DSSFlow selectFlowByID(Long id); - - List listFlowByTaxonomyID(@Param("projectID") Long projectID, @Param("taxonomyID") Long taxonomyID, @Param("isRootFlow") Boolean isRootFlow); - - List listFlowVersionsByFlowID(@Param("flowID") Long flowID, @Param("projectVersionID") Long projectVersionID); - - void insertFlow(DSSFlow dssFlow) throws DuplicateKeyException; - - void insertFlowVersion(DSSFlowVersion version); - - void batchInsertFlowVersion(@Param("flowVersions") List flowVersions); - - void insertFlowRelation(@Param("flowID") Long flowID, @Param("parentFlowID") Long parentFlowID); - - DSSFlowVersion selectVersionByFlowID(@Param("flowID") Long flowID, @Param("version") String version, @Param("projectVersionID") Long projectVersionID); - - void updateFlowBaseInfo(DSSFlow dssFlow) throws DuplicateKeyException; - - List selectSubFlowIDByParentFlowID(Long parentFlowID); - - void deleteFlowVersions(@Param("flowID") Long flowID, @Param("projectVersionID") Long projectVersionID); - - void deleteFlowBaseInfo(Long flowID); - - void deleteFlowRelation(Long flowID); - - Long selectParentFlowIDByFlowID(Long flowID); - - List listFlowByProjectID(Long projectID); - - List listVersionByFlowIDAndProjectVersionID(@Param("flowID") Long flowID, @Param("projectVersionID") Long projectVersionID); - - Boolean noVersions(Long flowID); - - List listLastFlowVersionsByProjectVersionID(@Param("projectVersionID") Long projectVersionId); - - List listLatestRootFlowVersionByProjectVersionID(Long projectVersionID); - - void batchUpdateFlowVersion(List flowVersions); - - Long getParentFlowID(Long flowID); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/FlowTaxonomyMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/FlowTaxonomyMapper.java deleted file mode 100644 index 640e8b7d0..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/FlowTaxonomyMapper.java +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - - -import com.webank.wedatasphere.dss.server.entity.DSSFlowTaxonomy; -import org.apache.ibatis.annotations.Param; -import org.springframework.dao.DuplicateKeyException; - -import java.util.List; - - -public interface FlowTaxonomyMapper { - DSSFlowTaxonomy selectFlowTaxonomyByID(Long id); - - void insertFlowTaxonomy(DSSFlowTaxonomy dssFlowTaxonomy) throws DuplicateKeyException; - - void updateFlowTaxonomy(DSSFlowTaxonomy dssFlowTaxonomy) throws DuplicateKeyException; - - Long hasFlows(Long flowTaxonomyID); - - void deleteFlowTaxonomy(Long flowTaxonomyID); - - - void insertFlowTaxonomyRelation(@Param("taxonomyID") Long taxonomyID, @Param("flowID") Long flowID); - - Long selectTaxonomyIDByFlowID(Long id); - - void updateFlowTaxonomyRelation(@Param("flowIDList") List flowIDList, @Param("taxonomyID") Long taxonomyID); - - void deleteFlowTaxonomyRelation(Long flowID); - - void deleteFlowTaxonomyByProjectID(Long projectID); - - List listFlowTaxonomyByProjectID(Long projectID); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/NodeInfoMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/NodeInfoMapper.java deleted file mode 100644 index dcf8d0594..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/NodeInfoMapper.java +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - - - - -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.server.entity.NodeInfo; - -import java.util.List; - - -public interface NodeInfoMapper { - List selectNodeInfo(String developProject); - - String getApplicationUrlByNodeType(String nodeType); - - Application getApplicationbyNodeType(String nodeType); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/ProjectMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/ProjectMapper.java deleted file mode 100644 index c2ae56786..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/ProjectMapper.java +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - - -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectPublishHistory; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectVersion; -import org.apache.ibatis.annotations.Param; - -import java.util.List; -import java.util.Map; - - -public interface ProjectMapper { - DSSProject selectProjectByID(Long id); - - DSSProjectVersion selectLatestVersionByProjectID(Long projectID); - - DSSProject selectProjectByVersionID(Long projectVersionID); - - void addProject(DSSProject dssProject); - - void addProjectVersion(DSSProjectVersion dssProjectVersion); - - void updateDescription(@Param("projectID") Long projectID, @Param("description") String description, @Param("product")String product ,@Param("applicationArea")Integer applicationArea ,@Param("business")String business); - - List listProjectVersionsByProjectID(Long projectID); - - Boolean noPublished(Long projectID); - - void deleteProjectVersions(long projectID); - - void deleteProjectBaseInfo(long projectID); - - DSSProjectVersion selectProjectVersionByID(Long id); - - DSSProjectVersion selectProjectVersionByProjectIDAndVersionID(@Param("projectID") Long projectId, @Param("version") String version); - - Integer updateLock(@Param("lock") Integer lock, @Param("projectVersionID") Long projectVersionID); - - DSSProjectPublishHistory selectProjectPublishHistoryByProjectVersionID(Long projectVersionID); - - void insertPublishHistory(DSSProjectPublishHistory dssProjectPublishHistory); - - void updatePublishHistoryState(@Param("projectVersionID") Long projectVersionID, @Param("status") Integer status); - - void addAccessProjectRelation(@Param("appjointProjectIDAndAppID") Map appjointProjectIDAndAppID, @Param("projectID") Long projectID); - - Long getAppjointProjectID(@Param("projectID") Long projectID, @Param("applicationID") Integer applicationID); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/ProjectTaxonomyMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/ProjectTaxonomyMapper.java deleted file mode 100644 index 21fc44e41..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/ProjectTaxonomyMapper.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - -import com.webank.wedatasphere.dss.server.entity.DSSProjectTaxonomy; -import com.webank.wedatasphere.dss.server.entity.DSSProjectTaxonomyRelation; -import org.apache.ibatis.annotations.Param; -import org.springframework.dao.DuplicateKeyException; - -import java.util.List; - - -public interface ProjectTaxonomyMapper { - DSSProjectTaxonomy selectProjectTaxonomyByID(Long id); - DSSProjectTaxonomyRelation selectProjectTaxonomyRelationByTaxonomyIdOrProjectId(Long taxonomyIdOrProjectId); - List listProjectTaxonomyByUser(String userName); - - //-------------------- - List listProjectIDByTaxonomyID(@Param("taxonomyID") Long taxonomyID, @Param("userName") String userName); - - void insertProjectTaxonomy(DSSProjectTaxonomy dssProjectTaxonomy) throws DuplicateKeyException; - - void updateProjectTaxonomy(DSSProjectTaxonomy dssProjectTaxonomy) throws DuplicateKeyException; - - Long hasProjects(Long projectTaxonomyID); - - void deleteProjectTaxonomy(Long projectTaxonomyID); - - void addProjectTaxonomyRelation(@Param("projectID") Long id, @Param("taxonomyID") Long taxonomyID, @Param("creatorID") Long creatorID); - - void updateProjectTaxonomyRelation(@Param("projectID") Long projectID, @Param("taxonomyID") Long taxonomyID); - - void deleteProjectTaxonomyRelationByProjectID(Long projectID); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/WorkspaceMapper.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/WorkspaceMapper.java deleted file mode 100644 index c3fbaca22..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/WorkspaceMapper.java +++ /dev/null @@ -1,68 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dao; - -import com.webank.wedatasphere.dss.server.dto.response.*; -import com.webank.wedatasphere.dss.server.entity.*; -import com.webank.wedatasphere.dss.server.dto.response.HomepageDemoInstanceVo; -import com.webank.wedatasphere.dss.server.dto.response.HomepageDemoMenuVo; -import com.webank.wedatasphere.dss.server.dto.response.HomepageVideoVo; -import com.webank.wedatasphere.dss.server.dto.response.WorkspaceFavoriteVo; -import org.apache.ibatis.annotations.Param; - -import java.util.List; - -/** - * Created by schumiyi on 2020/6/22 - */ -public interface WorkspaceMapper { - - List getWorkspaces(); - - List findByWorkspaceName(String name); - - void addWorkSpace(DSSWorkspace dssWorkspace); - - List getHomepageDemoMenusEn(); - List getHomepageDemoMenusCn(); - - List getHomepageInstancesByMenuIdCn(Long id); - List getHomepageInstancesByMenuIdEn(Long id); - - List getHomepageVideosEn(); - List getHomepageVideosCn(); - - DSSWorkspace getWorkspaceById(Long workspaceId); - - List getManagementMenuCn(); - List getManagementMenuEn(); - - List getApplicationMenuCn(); - List getApplicationMenuEn(); - - List getMenuAppInstancesCn(Long id); - List getMenuAppInstancesEn(Long id); - - List getWorkspaceFavoritesCn(@Param("username") String username, @Param("workspaceId") Long workspaceId); - - List getWorkspaceFavoritesEn(@Param("username") String username, @Param("workspaceId") Long workspaceId); - - void addFavorite(DSSFavorite dssFavorite); - - void deleteFavorite(Long favouritesId); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/dwsUserMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/dwsUserMapper.xml deleted file mode 100644 index ac65dce4f..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/dwsUserMapper.xml +++ /dev/null @@ -1,31 +0,0 @@ - - - - - - - - - - - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/flowMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/flowMapper.xml deleted file mode 100644 index 37dae091a..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/flowMapper.xml +++ /dev/null @@ -1,256 +0,0 @@ - - - - - - - - - - id,`name`,`state`,`source`,`description`,`create_time`,`creator_id`,`is_root_flow`,`rank`,`project_id`,`has_saved`,`uses` - - - - id,`flow_id`,`source`,`version`,`json_path`,`comment`,`update_time`,`updator_id`,`project_version_id` - - - - - - - - - - - - - - - - INSERT INTO dss_flow () - VALUES - (#{id},#{name},#{state},#{source},#{description},#{createTime},#{creatorID},#{isRootFlow},#{rank},#{projectID},#{hasSaved},#{uses}) - - - - INSERT INTO dss_flow_version () - VALUES - (#{id},#{flowID},#{source},#{version},#{jsonPath},#{comment},#{updateTime},#{updatorID},#{projectVersionID}) - - - - INSERT INTO dss_flow_version - ( `flow_id`,`source`,`version`,`json_path`,`comment`,`update_time`,`updator_id`,`project_version_id`) - VALUES - - (#{fv.flowID},#{fv.source},#{fv.version},#{fv.jsonPath},#{fv.comment},#{fv.updateTime},#{fv.updatorID},#{fv.projectVersionID}) - - - - - - UPDATE dss_flow_version - source= #{fv.source} - version= #{fv.version} - json_path= #{fv.jsonPath} - comment= #{fv.comment} - update_time= #{fv.updateTime} - updator_id= #{fv.updatorID} - project_version_id= #{fv.projectVersionID} - flow_id = #{fv.flowID} - - - - - INSERT INTO dss_flow_relation (flow_id,parent_flow_id) - VALUES - (#{flowID},#{parentFlowID}) - - - - - - UPDATE dss_flow - - name=#{name}, - description=#{description}, - has_saved=#{hasSaved}, - uses=#{uses}, - - WHERE id =#{id} - - - - - - DELETE - FROM - dss_flow_version - WHERE flow_id = #{flowID} AND project_version_id = #{projectVersionID} - - - - DELETE - FROM - dss_flow - WHERE id = #{flowID} - - - - DELETE - FROM - dss_flow_relation - WHERE flow_id = #{flowID} - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/flowTaxonomyMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/flowTaxonomyMapper.xml deleted file mode 100644 index c681f768c..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/flowTaxonomyMapper.xml +++ /dev/null @@ -1,95 +0,0 @@ - - - - - - - - - - id, `name`,`description`,`creator_id`,`create_time`,`update_time`,`project_id` - - - - - - INSERT INTO dss_flow_taxonomy () - VALUES - (#{id},#{name},#{description},#{creatorID},#{createTime},#{updateTime},#{projectID}) - - - - UPDATE dss_flow_taxonomy - - name=#{name}, - description=#{description}, - update_time=#{updateTime}, - - WHERE id =#{id} - - - - - - DELETE FROM dss_flow_taxonomy WHERE id = #{flowTaxonomyID} - - - - INSERT INTO dss_flow_taxonomy_relation(taxonomy_id,flow_id) VALUES (#{taxonomyID},#{flowID}) - - - - - - UPDATE dss_flow_taxonomy_relation - SET taxonomy_id = #{taxonomyID} - WHERE #{element} - - - - DELETE - FROM - dss_flow_taxonomy_relation - WHERE flow_id = #{flowID} - - - - DELETE - FROM - `dss_flow_taxonomy` - WHERE - project_id = #{projectID} - - - - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/nodeInfoMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/nodeInfoMapper.xml deleted file mode 100644 index 2da460830..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/nodeInfoMapper.xml +++ /dev/null @@ -1,52 +0,0 @@ - - - - - - - - - - - - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/projectMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/projectMapper.xml deleted file mode 100644 index a1bc4ab8c..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/projectMapper.xml +++ /dev/null @@ -1,177 +0,0 @@ - - - - - - - - - - id,`name`,`source`,`description`,`org_id`,`visibility`,`is_transfer`,`initial_org_id`,`user_id`,`create_time`,`create_by`,`product`,`application_area`,`business`,`workspace_id` - - - - id,`project_id`,`version`,`comment`,`update_time`,`updator_id`,`lock` - - - - id,`project_version_id`,`create_time`,`creator_id`,`update_time`,`comment`,`state`,`version_path`,`expire_time` - - - - - - - - - - INSERT INTO dss_project () - VALUES - (#{id},#{name},#{source},#{description},#{orgID},#{visibility},#{isTransfer},#{initialOrgID},#{userID},#{createTime},#{createBy},#{product},#{applicationArea},#{business},#{workspaceId}) - - - - INSERT INTO dss_project_version () - VALUES - (#{id},#{projectID},#{version},#{comment},#{updateTime},#{updatorID},#{lock}) - - - - UPDATE dss_project - SET description = #{description},product = #{product},application_area = #{applicationArea},business = #{business} - WHERE id = #{projectID} - - - - - - - - - - DELETE - FROM - dss_project_version - WHERE - project_id = #{projectID} - - - - DELETE - FROM - dss_project - WHERE - id = #{projectID} - - - - - - - - UPDATE `dss_project_version` set `lock` = `lock` +1 WHERE id= #{projectVersionID} AND `lock` = #{lock} - - - - - - INSERT INTO dss_project_publish_history () - VALUES - (#{id},#{projectVersionID},#{createTime},#{createID},#{updateTime},#{comment},#{state},#{versionPath},#{expireTime}) - - - - UPDATE `dss_project_publish_history` - SET state = #{status} - WHERE - project_version_id = #{projectVersionID} - - - - INSERT INTO dss_project_applications_project - ( `project_id`,`application_id`,`application_project_id`) - VALUES - - (#{projectID},#{key},#{item}) - - - - - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/projectTaxonomyMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/projectTaxonomyMapper.xml deleted file mode 100644 index 9c0333953..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/projectTaxonomyMapper.xml +++ /dev/null @@ -1,97 +0,0 @@ - - - - - - - - - id, `name`,`description`,`creator_id`,`create_time`,`update_time` - - - - - - - - - - - INSERT INTO dss_project_taxonomy () - VALUES - (#{id},#{name},#{description},#{creatorID},#{createTime},#{updateTime}) - - - - UPDATE dss_project_taxonomy - - name=#{name}, - description=#{description}, - update_time=#{updateTime}, - - WHERE id =#{id} - - - - - - DELETE FROM dss_project_taxonomy WHERE id = #{projectTaxonomyID} - - - - INSERT INTO dss_project_taxonomy_relation VALUES (#{taxonomyID},#{projectID},#{creatorID}) - - - - UPDATE dss_project_taxonomy_relation set taxonomy_id = #{taxonomyID} WHERE project_id = #{projectID} - - - - DELETE - FROM - dss_project_taxonomy_relation - WHERE - project_id = #{projectID} - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/workspaceMapper.xml b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/workspaceMapper.xml deleted file mode 100644 index e1fc05ed9..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dao/impl/workspaceMapper.xml +++ /dev/null @@ -1,200 +0,0 @@ - - - - - - - - - id,`name`,`label`,`description`,`department`,`product`,`source`,`create_by`,`create_time`,`last_update_user`,`last_update_time` - - - - m.`id`,m.`title_cn` AS `title`, m.`desc_cn` AS `description`,m.`labels_cn` AS `labels`, - m.`access_button_cn` AS `access_button`,m.`manual_button_cn` AS `manualButton`,m.`is_active`, - m.`manual_button_url`,m.`icon`,m.`order`,app.`homepage_url` AS `access_button_url`,app.project_url, app.`name` - - - m.`id`,m.`title_en` AS `title`, m.`desc_en` AS `description`,m.`labels_en` AS `labels`, - m.`access_button_en` AS `access_button`,m.`manual_button_en` AS `manualButton`,m.`is_active`, - m.`manual_button_url`,m.`icon`,m.`order`,app.`homepage_url` AS `access_button_url`,app.project_url, app.`name` - - - - id,`username`,`workspace_id`,`menu_application_id`,`order`,`create_by`,`create_time`,`last_update_user`,`last_update_time` - - - - - - - - - - INSERT INTO dss_workspace () - VALUES - (#{id},#{name},#{label},#{description},#{department},#{product},#{source},#{createBy},#{createTime},#{lastUpdateUser},#{lastUpdateTime}) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - INSERT INTO dss_onestop_user_favorites () - VALUES - (#{id},#{username},#{workspaceId},#{menuApplicationId},#{order},#{createBy},#{createTime},#{lastUpdateUser},#{lastUpdateTime}) - - - - DELETE - FROM - dss_onestop_user_favorites - WHERE - id = #{favouritesId} - - \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/db/DSSMybatisConfig.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/db/DSSMybatisConfig.java deleted file mode 100644 index 2ce87d062..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/db/DSSMybatisConfig.java +++ /dev/null @@ -1,148 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.db; - -import com.webank.wedatasphere.dss.server.conf.DSSServerConf; -import org.apache.commons.collections.CollectionUtils; -import org.apache.commons.dbcp.BasicDataSource; -import org.apache.commons.lang.StringUtils; -import org.apache.ibatis.session.SqlSessionFactory; -import org.mybatis.spring.SqlSessionFactoryBean; -import org.mybatis.spring.mapper.MapperScannerConfigurer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Qualifier; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; -import org.springframework.context.annotation.Primary; -import org.springframework.core.io.DefaultResourceLoader; -import org.springframework.core.io.Resource; -import org.springframework.core.io.support.PathMatchingResourcePatternResolver; -import org.springframework.jdbc.datasource.DataSourceTransactionManager; -import org.springframework.transaction.PlatformTransactionManager; -import org.springframework.transaction.annotation.EnableTransactionManagement; - -import javax.sql.DataSource; -import java.io.IOException; -import java.util.ArrayList; -import java.util.List; - - -@Configuration -@EnableTransactionManagement(order = 2) -public class DSSMybatisConfig { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - - @Bean(name = "dataSource", destroyMethod = "close") - @Primary - public DataSource dataSource() { - BasicDataSource datasource = new BasicDataSource(); - String dbUrl = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_URL.getValue(); - String username = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_USERNAME.getValue(); - String password = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_PASSWORD.getValue(); - String driverClassName = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_DRIVER_CLASS_NAME.getValue(); - int initialSize = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_INITIALSIZE.getValue(); - int minIdle = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_MINIDLE.getValue(); - int maxActive = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_MAXACTIVE.getValue(); - int maxWait = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_MAXWAIT.getValue(); - int timeBetweenEvictionRunsMillis = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_TBERM.getValue(); - int minEvictableIdleTimeMillis = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_MEITM.getValue(); - String validationQuery = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_VALIDATIONQUERY.getValue(); - boolean testWhileIdle = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_TESTWHILEIDLE.getValue(); - boolean testOnBorrow = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_TESTONBORROW.getValue(); - boolean testOnReturn = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_TESTONRETURN.getValue(); - boolean poolPreparedStatements = DSSServerConf.BDP_SERVER_MYBATIS_DATASOURCE_POOLPREPAREDSTATEMENTS.getValue(); - datasource.setUrl(dbUrl); - datasource.setUsername(username); - datasource.setPassword(password); - datasource.setDriverClassName(driverClassName); - datasource.setInitialSize(initialSize); - datasource.setMinIdle(minIdle); - datasource.setMaxActive(maxActive); - datasource.setMaxWait(maxWait); - datasource.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis); - datasource.setMinEvictableIdleTimeMillis(minEvictableIdleTimeMillis); - datasource.setValidationQuery(validationQuery); - datasource.setTestWhileIdle(testWhileIdle); - datasource.setTestOnBorrow(testOnBorrow); - datasource.setTestOnReturn(testOnReturn); - datasource.setPoolPreparedStatements(poolPreparedStatements); - logger.info("Database connection address information(数据库连接地址信息)=" + dbUrl); - return datasource; - } - - @Bean(name = "sqlSessionFactory") - @Primary - public SqlSessionFactory sqlSessionFactory(@Qualifier("dataSource") DataSource dataSource) throws Exception { - String typeAliasesPackage = DSSServerConf.BDP_SERVER_MYBATIS_TYPEALIASESPACKAGE.getValue(); - //Configure the mapper scan to find all mapper.xml mapping files(配置mapper的扫描，找到所有的mapper.xml映射文件) - String mapperLocations = DSSServerConf.BDP_SERVER_MYBATIS_MAPPER_LOCATIONS.getValue(); - //Load the global configuration file(加载全局的配置文件) - String configLocation = DSSServerConf.BDP_SERVER_MYBATIS_CONFIGLOCATION.getValue(); - try { - SqlSessionFactoryBean sessionFactoryBean = new SqlSessionFactoryBean(); - sessionFactoryBean.setDataSource(dataSource); - - logger.info("Mybatis typeAliasesPackage=" + typeAliasesPackage); - logger.info("Mybatis mapperLocations=" + mapperLocations); - logger.info("Mybatis configLocation=" + configLocation); - // Read configuration(读取配置) - sessionFactoryBean.setTypeAliasesPackage(typeAliasesPackage); - - //Set the location of the mapper.xml file(设置mapper.xml文件所在位置) - if(StringUtils.isNotBlank(mapperLocations)) { - String[] mapperArray = mapperLocations.split(","); - List resources = new ArrayList<>(); - for(String mapperLocation : mapperArray){ - CollectionUtils.addAll(resources,new PathMatchingResourcePatternResolver().getResources(mapperLocation)); - } - sessionFactoryBean.setMapperLocations(resources.toArray(new Resource[0])); - } - /* Resource[] resources = new PathMatchingResourcePatternResolver().getResources(mapperLocations); - sessionFactoryBean.setMapperLocations(resources);*/ -// Set the location of the mybatis-config.xml configuration file(设置mybatis-config.xml配置文件位置) - sessionFactoryBean.setConfigLocation(new DefaultResourceLoader().getResource(configLocation)); - - return sessionFactoryBean.getObject(); - } catch (IOException e) { - logger.error("mybatis resolver mapper*xml is error",e); - return null; - } catch (Exception e) { - logger.error("mybatis sqlSessionFactoryBean create error",e); - return null; - } - } - - @Primary - public PlatformTransactionManager annotationDrivenTransactionManager(@Qualifier("dataSource") DataSource dataSource) { - return new DataSourceTransactionManager(dataSource); - } - - @Bean - public MapperScannerConfigurer mapperScannerConfigurer(@Qualifier("sqlSessionFactory") SqlSessionFactory sqlSessionFactory) { - MapperScannerConfigurer mapperScannerConfigurer = new MapperScannerConfigurer(); - mapperScannerConfigurer.setSqlSessionFactory(sqlSessionFactory); - //Each table corresponds to the XXMapper.java interface type Java file - //每张表对应的XXMapper.java interface类型的Java文件 - mapperScannerConfigurer.setBasePackage(DSSServerConf.BDP_SERVER_MYBATIS_BASEPACKAGE.getValue()); - return mapperScannerConfigurer; - } - -} \ No newline at end of file diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/WorkspaceFavoriteVo.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/WorkspaceFavoriteVo.java deleted file mode 100644 index f9953d6c5..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/dto/response/WorkspaceFavoriteVo.java +++ /dev/null @@ -1,82 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.dto.response; - -/** - * Created by Adamyuanyuan on 2020/6/25 - */ -public class WorkspaceFavoriteVo { - private Long id; - - private Long menuApplicationId; - - private String name; - - private String url; - - private String icon; - - private String title; - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public Long getMenuApplicationId() { - return menuApplicationId; - } - - public void setMenuApplicationId(Long menuApplicationId) { - this.menuApplicationId = menuApplicationId; - } - - public String getUrl() { - return url; - } - - public void setUrl(String url) { - this.url = url; - } - - public String getIcon() { - return icon; - } - - public void setIcon(String icon) { - this.icon = icon; - } - - public String getTitle() { - return title; - } - - public void setTitle(String title) { - this.title = title; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/BaseEntity.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/BaseEntity.java deleted file mode 100644 index d01380d9e..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/BaseEntity.java +++ /dev/null @@ -1,49 +0,0 @@ -package com.webank.wedatasphere.dss.server.entity; - -import java.util.Date; - -/** - * Created by schumiyi on 2020/6/22 - */ -public class BaseEntity { - - private String createBy; - - private Date createTime = new Date(); - - private Date lastUpdateTime = new Date(); - - private String lastUpdateUser; - - public String getCreateBy() { - return createBy; - } - - public void setCreateBy(String createBy) { - this.createBy = createBy; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Date getLastUpdateTime() { - return lastUpdateTime; - } - - public void setLastUpdateTime(Date lastUpdateTime) { - this.lastUpdateTime = lastUpdateTime; - } - - public String getLastUpdateUser() { - return lastUpdateUser; - } - - public void setLastUpdateUser(String lastUpdateUser) { - this.lastUpdateUser = lastUpdateUser; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSFlowTaxonomy.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSFlowTaxonomy.java deleted file mode 100644 index 314e21ea5..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSFlowTaxonomy.java +++ /dev/null @@ -1,99 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.entity; - -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; - -import java.util.Date; -import java.util.List; - - -public class DSSFlowTaxonomy { - private Long id; - private String name; - private String description; - private Long creatorID; - private Date createTime; - private Date updateTime; - private Long projectID; - private List dssFlowList; - - public List getDssFlowList() { - return dssFlowList; - } - - public void setDssFlowList(List dssFlowList) { - this.dssFlowList = dssFlowList; - } - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public Long getCreatorID() { - return creatorID; - } - - public void setCreatorID(Long creatorID) { - this.creatorID = creatorID; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Long getProjectID() { - return projectID; - } - - public void setProjectID(Long projectID) { - this.projectID = projectID; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSProjectTaxonomy.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSProjectTaxonomy.java deleted file mode 100644 index 302b80675..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSProjectTaxonomy.java +++ /dev/null @@ -1,90 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.entity; - -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; - -import java.util.Date; -import java.util.List; - - -public class DSSProjectTaxonomy { - private Long id; - private String name; - private String description; - private Long creatorID; - private Date createTime; - private Date updateTime; - private List dssProjectList; - - public List getDssProjectList() { - return dssProjectList; - } - - public void setDssProjectList(List dssProjectList) { - this.dssProjectList = dssProjectList; - } - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public Long getCreatorID() { - return creatorID; - } - - public void setCreatorID(Long creatorID) { - this.creatorID = creatorID; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSProjectTaxonomyRelation.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSProjectTaxonomyRelation.java deleted file mode 100644 index accdf3e4b..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSProjectTaxonomyRelation.java +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.entity; - -public class DSSProjectTaxonomyRelation { - private Long taxonomyId; - private Long projectId; - private Long creatorId; - - - public Long getTaxonomyId() { - return taxonomyId; - } - - public void setTaxonomyId(Long taxonomyId) { - this.taxonomyId = taxonomyId; - } - - public Long getProjectId() { - return projectId; - } - - public void setProjectId(Long projectId) { - this.projectId = projectId; - } - - public Long getCreatorId() { - return creatorId; - } - - public void setCreatorId(Long creatorId) { - this.creatorId = creatorId; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSWorkspace.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSWorkspace.java deleted file mode 100644 index 305bffeb4..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/DSSWorkspace.java +++ /dev/null @@ -1,93 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package com.webank.wedatasphere.dss.server.entity; - -/** - * Created by schumiyi on 2020/6/22 - */ -public class DSSWorkspace extends BaseEntity { - - private Long id; - - private String name; - - private String label; - - private String description; - - private String department; - - private String product; - - private String source; - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getName() { - return name; - } - - public void setName(String name) { - this.name = name; - } - - public String getLabel() { - return label; - } - - public void setLabel(String label) { - this.label = label; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public String getDepartment() { - return department; - } - - public void setDepartment(String department) { - this.department = department; - } - - public String getProduct() { - return product; - } - - public void setProduct(String product) { - this.product = product; - } - - public String getSource() { - return source; - } - - public void setSource(String source) { - this.source = source; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/NodeInfo.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/NodeInfo.java deleted file mode 100644 index 4ce3c6620..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/entity/NodeInfo.java +++ /dev/null @@ -1,103 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.entity; - - -public class NodeInfo { - private Integer id; - private String icon; - private String nodeType; - private Integer applicationId; - private Boolean submitToScheduler; //是否可以提交执行 - private Boolean enableCopy; - private Boolean shouldCreationBeforeNode; - private Boolean supportJump; - private String jumpUrl; - - public Integer getId() { - return id; - } - - public void setId(Integer id) { - this.id = id; - } - - public String getIcon() { - return icon; - } - - public void setIcon(String icon) { - this.icon = icon; - } - - public String getNodeType() { - return nodeType; - } - - public void setNodeType(String nodeType) { - this.nodeType = nodeType; - } - - public Integer getApplicationId() { - return applicationId; - } - - public void setApplicationId(Integer applicationId) { - this.applicationId = applicationId; - } - - public Boolean getSubmitToScheduler() { - return submitToScheduler; - } - - public void setSubmitToScheduler(Boolean submitToScheduler) { - this.submitToScheduler = submitToScheduler; - } - - public Boolean getEnableCopy() { - return enableCopy; - } - - public void setEnableCopy(Boolean enableCopy) { - this.enableCopy = enableCopy; - } - - public Boolean getShouldCreationBeforeNode() { - return shouldCreationBeforeNode; - } - - public void setShouldCreationBeforeNode(Boolean shouldCreationBeforeNode) { - this.shouldCreationBeforeNode = shouldCreationBeforeNode; - } - - public Boolean getSupportJump() { - return supportJump; - } - - public void setSupportJump(Boolean supportJump) { - this.supportJump = supportJump; - } - - public String getJumpUrl() { - return jumpUrl; - } - - public void setJumpUrl(String jumpUrl) { - this.jumpUrl = jumpUrl; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/FunctionInvoker.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/FunctionInvoker.java deleted file mode 100644 index d4ef8bc4b..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/FunctionInvoker.java +++ /dev/null @@ -1,104 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.function; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.service.NodeService; -import com.webank.wedatasphere.dss.appjoint.service.SecurityService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.Project; -import com.webank.wedatasphere.dss.server.dao.ProjectMapper; -import org.apache.commons.math3.util.Pair; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import java.util.ArrayList; -import java.util.List; -import java.util.Map; - - -@Component -public class FunctionInvoker { - - @Autowired - private ApplicationService applicationService; - @Autowired - private ProjectMapper projectMapper; - - private final Logger logger = LoggerFactory.getLogger(this.getClass()); - - public Map nodeServiceFunction(String userName, Map requestBody, AppJointNode node, NodeServiceFunction function) throws AppJointErrorException { - AppJoint appJoint = applicationService.getAppjoint(node.getNodeType()); - SecurityService securityService = appJoint.getSecurityService(); - Session session = null; - Map jobContent = null; - if(securityService != null){ - logger.info("appjoint securityService is exist,{}login",userName); - session = securityService.login(userName); - } - NodeService nodeService = appJoint.getNodeService(); - if(nodeService != null){ - logger.info("appJoint NodeService is exist"); - jobContent = function.accept(nodeService,session, node, requestBody); - } - return jobContent; - } - - public List> projectServiceAddFunction(DSSProject project, ProjectServiceAddFunction projectServiceAddFunction, List appJoints) throws AppJointErrorException { - ArrayList> projects = new ArrayList<>(); - for (AppJoint appJoint : appJoints) { - Project appJointProject = null; - Session session = null; - if (appJoint.getSecurityService() != null) { - logger.info("[addProject]securityService is exist,{}login...", project.getUserName()); - session = appJoint.getSecurityService().login(project.getUserName()); - } - if (appJoint.getProjectService() != null) { - logger.info("[addProject]projectService is exist"); - appJointProject = projectServiceAddFunction.accept(appJoint.getProjectService(), project, session); - if(appJointProject != null) projects.add(new Pair(appJointProject,appJoint.getAppJointName())); - } - } - return projects; - } - - public void projectServiceFunction(DSSProject project, ProjectServiceFunction projectServiceFunction, List appJoints) throws AppJointErrorException { - for (AppJoint appJoint : appJoints) { - Session session = null; - if(appJoint.getSecurityService() != null){ - logger.info("securityService is exist,{}login..",project.getUserName()); - session = appJoint.getSecurityService().login(project.getUserName()); - } - //通过projectID获取到appjointID - Application application = applicationService.getApplication(appJoint.getAppJointName()); - Long appjointProjectID = projectMapper.getAppjointProjectID(project.getId(), application.getId()); - if(appjointProjectID != null) project.setId(appjointProjectID); - if(appJoint.getProjectService() != null){ - logger.info("projectService is exist"); - projectServiceFunction.accept(appJoint.getProjectService(),project,session); - } - } - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/FunctionPool.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/FunctionPool.java deleted file mode 100644 index 7efe477b2..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/FunctionPool.java +++ /dev/null @@ -1,39 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ -package com.webank.wedatasphere.dss.server.function; - -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.service.NodeService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - -import java.util.Map; - - -public class FunctionPool { - - public static NodeServiceFunction deleteNode = (NodeService nodeService, Session session, AppJointNode node, Map requestBody)->{ - node.setJobContent(requestBody); - nodeService.deleteNode(session,node); - return null; - }; - - public static NodeServiceFunction createNode = NodeService::createNode; - - public static NodeServiceFunction updateNode = NodeService::updateNode; - - // TODO: 2019/11/26 把projectService也纳入 -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/NodeServiceFunction.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/NodeServiceFunction.java deleted file mode 100644 index 71873f0de..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/NodeServiceFunction.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.function; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.service.NodeService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - -import java.util.Map; - - -public interface NodeServiceFunction{ - Map accept(NodeService nodeService, Session session, AppJointNode node, Map requestBody) throws AppJointErrorException; -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/ProjectServiceAddFunction.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/ProjectServiceAddFunction.java deleted file mode 100644 index b5eb0b322..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/ProjectServiceAddFunction.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.function; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.project.Project; - - -public interface ProjectServiceAddFunction { - Project accept(ProjectService projectService, Project project, Session session) throws AppJointErrorException; -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/ProjectServiceFunction.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/ProjectServiceFunction.java deleted file mode 100644 index b79e83bd4..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/function/ProjectServiceFunction.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.function; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.project.Project; - - -public interface ProjectServiceFunction { - void accept(ProjectService projectService, Project project, Session session) throws AppJointErrorException; -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/LockAspect.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/LockAspect.java deleted file mode 100644 index 473f224c5..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/lock/LockAspect.java +++ /dev/null @@ -1,110 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.lock; - -import com.webank.wedatasphere.dss.server.constant.DSSServerConstant; -import com.webank.wedatasphere.dss.server.dao.ProjectMapper; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectVersion; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.apache.commons.lang.ArrayUtils; -import org.aspectj.lang.ProceedingJoinPoint; -import org.aspectj.lang.Signature; -import org.aspectj.lang.annotation.Around; -import org.aspectj.lang.annotation.Aspect; -import org.aspectj.lang.annotation.Pointcut; -import org.aspectj.lang.reflect.MethodSignature; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.core.Ordered; -import org.springframework.stereotype.Component; - -import java.lang.reflect.Method; -import java.util.List; - - -@Aspect -@Component -public class LockAspect implements Ordered { - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private ProjectMapper projectMapper; - - @Pointcut(value = "@annotation(com.webank.wedatasphere.dss.server.lock.Lock)") - private void cut() { - } - - @Around("cut()") - public Object around(ProceedingJoinPoint point) throws Throwable { - Signature signature = point.getSignature(); - if (!(signature instanceof MethodSignature)) { - throw new IllegalArgumentException("该注解只能用于方法"); - } - MethodSignature methodSignature = (MethodSignature) signature; - Object target = point.getTarget(); - Method currentMethod = target.getClass().getMethod(methodSignature.getName(), methodSignature.getParameterTypes()); - Lock lockAnnotation = currentMethod.getAnnotation(Lock.class); - logger.info("调用方法：" + currentMethod.getName()); - Object[] args = point.getArgs(); - String[] parameterNames = methodSignature.getParameterNames(); - int index = ArrayUtils.indexOf(parameterNames, DSSServerConstant.PROJECT_VERSION_ID); - Long projectVersionID = (Long) args[index]; - if (projectVersionID == null) { - logger.info("执行删除工程，直接通过"); - return point.proceed(); - } - logger.info("projectVersionID为：" + projectVersionID); - DSSProjectVersion dssProjectVersion = projectMapper.selectProjectVersionByID(projectVersionID); - Integer lock = dssProjectVersion.getLock(); - try { - Object proceed = point.proceed(); - judge(lockAnnotation, dssProjectVersion, lock, projectVersionID); - return proceed; - } catch (Exception e) { - logger.info("执行过程出现异常", e); - throw e; - } - } - - private void judge(Lock lockAnnotation, DSSProjectVersion dssProjectVersion, Integer lock, Long projectVersionID) throws DSSErrorException { - if (lockAnnotation.type().equals(LockEnum.ADD)) { - logger.info("projectVersion会增加"); - List dssProjectVersions = projectMapper.listProjectVersionsByProjectID(dssProjectVersion.getProjectID()); - if (dssProjectVersions.size() < 2 || !dssProjectVersions.get(1).getId().equals(projectVersionID)) { - throw new DSSErrorException(67457, "已经有别的用户对此project进行了版本更新操作，不能进行此操作！"); - } - } else { - logger.info("projectVersion不会增加"); - DSSProjectVersion latest = projectMapper.selectLatestVersionByProjectID(dssProjectVersion.getProjectID()); - if (!latest.getId().equals(projectVersionID)) { - throw new DSSErrorException(67455, "目前project版本已经不是最新，不能进行此操作！"); - } - } - Integer row = projectMapper.updateLock(lock, projectVersionID); - if (row != 1) { - throw new DSSErrorException(67456, "已经有别的用户对此project进行了更新，不能进行此操作！"); - } - } - - - @Override - public int getOrder() { - return 3; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/AppJointNodeOperate.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/AppJointNodeOperate.java deleted file mode 100644 index 276e99d3e..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/AppJointNodeOperate.java +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.operate; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.core.CommonAppJointNode; -import com.webank.wedatasphere.dss.server.function.FunctionInvoker; -import com.webank.wedatasphere.dss.server.function.FunctionPool; -import com.webank.wedatasphere.dss.server.function.NodeServiceFunction; -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - - -@Component -public class AppJointNodeOperate implements Operate { - - @Autowired - private FunctionInvoker functionInvoker; - - @Override - public boolean canInvokeOperate(Op op) { - return !"workflow.subflow".equals(op.getNodeType()); - } - - @Override - public void add(DSSFlowService dssFlowService, Op op) throws AppJointErrorException { - invokeNodeServiceFunction(op,FunctionPool.createNode); - } - - @Override - public void update(DSSFlowService dssFlowService, Op op) throws AppJointErrorException { - invokeNodeServiceFunction(op,FunctionPool.updateNode); - } - - @Override - public void delete(DSSFlowService dssFlowService, Op op) throws AppJointErrorException { - invokeNodeServiceFunction(op, FunctionPool.deleteNode); - } - - private void invokeNodeServiceFunction(Op op,NodeServiceFunction function) throws AppJointErrorException { - String userName = op.getParams().get("userName").toString(); - Long projectID = Long.valueOf(op.getParams().get("projectID").toString()); - op.getParams().put("id",op.getId()); - CommonAppJointNode node = new CommonAppJointNode(); - node.setProjectId(projectID); - node.setNodeType(op.getNodeType()); - functionInvoker.nodeServiceFunction(userName,op.getParams(),node,function); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/Op.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/Op.java deleted file mode 100644 index 6ee9c2dd5..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/Op.java +++ /dev/null @@ -1,60 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.operate; - -import java.util.Map; - - -public class Op { - private Long id; - private String op; - private String nodeType; - private Map params; - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getNodeType() { - return nodeType; - } - - public void setNodeType(String nodeType) { - this.nodeType = nodeType; - } - - public String getOp() { - return op; - } - - public void setOp(String op) { - this.op = op; - } - - public Map getParams() { - return params; - } - - public void setParams(Map params) { - this.params = params; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/Operate.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/Operate.java deleted file mode 100644 index ec44935c8..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/Operate.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.operate; - - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - - -public interface Operate { - - boolean canInvokeOperate(Op op); - - void add(DSSFlowService dssFlowService, Op op) throws DSSErrorException, AppJointErrorException; - - void update(DSSFlowService dssFlowService, Op op) throws DSSErrorException, AppJointErrorException; - - void delete(DSSFlowService dssFlowService, Op op) throws DSSErrorException, AppJointErrorException; -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/SubFlowOperate.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/SubFlowOperate.java deleted file mode 100644 index 8f3d28661..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/operate/SubFlowOperate.java +++ /dev/null @@ -1,73 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.operate; - - -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.stereotype.Component; - -import java.util.Arrays; - - -@Component -public class SubFlowOperate implements Operate { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - private String nodeType = "workflow.subflow"; - - @Override - public boolean canInvokeOperate(Op op) { - return this.nodeType.equals(op.getNodeType()); - } - - @Override - public void add(DSSFlowService dssFlowService, Op op)throws DSSErrorException { - afterOperateSubFlow(dssFlowService,op); - } - - @Override - public void update(DSSFlowService dssFlowService, Op op) throws DSSErrorException { - logger.info("name:{},description:{}",op.getParams().get("name"),op.getParams().get("description")); - DSSFlow dssFlow = new DSSFlow(); - dssFlow.setId(op.getId()); - dssFlow.setName(op.getParams().get("name").toString()); - dssFlow.setDescription(op.getParams().get("description").toString()); - dssFlowService.updateFlowBaseInfo(dssFlow, Long.valueOf(op.getParams().get("projectVersionID").toString()), null); - afterOperateSubFlow(dssFlowService,op); - } - - @Override - public void delete(DSSFlowService dssFlowService, Op op) throws DSSErrorException { - logger.info("delete subFlow{}",op.getId()); - dssFlowService.batchDeleteFlow(Arrays.asList(op.getId()), Long.valueOf(op.getParams().get("projectVersionID").toString())); - afterOperateSubFlow(dssFlowService,op); - } - - private void afterOperateSubFlow(DSSFlowService dssFlowService, Op op) throws DSSErrorException { - //更新工作流基本信息 - DSSFlow dssFlow = new DSSFlow(); - dssFlow.setId(op.getId()); - dssFlow.setHasSaved(true); - dssFlowService.updateFlowBaseInfo(dssFlow, Long.valueOf(op.getParams().get("projectVersionID").toString()), null); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJob.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJob.java deleted file mode 100644 index d324b6fa3..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJob.java +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - - -public abstract class PublishJob implements Runnable { - protected Logger logger = LoggerFactory.getLogger(this.getClass()); - protected String user; - protected Long projectVersionID; - protected PublishListner publishListner; - - public void setPublishListner(PublishListner publishListner) { - this.publishListner = publishListner; - } - - public PublishListner getPublishListner() { - return publishListner; - } - - public void setProjectVersionID(Long projectVersionID) { - this.projectVersionID = projectVersionID; - } - - public Long getProjectVersionID() { - return projectVersionID; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJobEnum.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJobEnum.java deleted file mode 100644 index c971b600e..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJobEnum.java +++ /dev/null @@ -1,23 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -public enum PublishJobEnum { - Inited,Running,Succeed,Failed -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJobFactory.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJobFactory.java deleted file mode 100644 index fff485910..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishJobFactory.java +++ /dev/null @@ -1,47 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -import com.webank.wedatasphere.dss.server.conf.DSSServerConf; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import java.util.concurrent.Future; - - -@Component -public class PublishJobFactory { - @Autowired - private PublishManager publishManager; - @Autowired - private DSSProjectService projectService; - - public PublishSubmitJob createSubmitPublishJob(Long projectVersionID, String user, String comment){ - PublishSubmitJob job = new PublishSubmitJob(projectService, user, comment, projectVersionID); - job.setPublishListner(publishManager); - return job; - } - - public PublishSubmitJobDeamon createSubmitPublishJobDeamon(Future future,PublishSubmitJob publishSubmitJob){ - PublishSubmitJobDeamon job = new PublishSubmitJobDeamon(future, (int) DSSServerConf.PUBLISH_TIMEOUT.getValue()); - job.setProjectVersionID(publishSubmitJob.getProjectVersionID()); - return job; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishListner.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishListner.java deleted file mode 100644 index 8a8cc447a..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishListner.java +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -public interface PublishListner { - - void onPublishSucceed(Long projectVersionID, String infoMsg); - - void onPublishFailed(Long projectVersionID, String errorMsg); - - void onPublishRunning(Long projectVersionID, String infoMsg); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishManager.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishManager.java deleted file mode 100644 index c1f29c4e2..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishManager.java +++ /dev/null @@ -1,166 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectPublishHistory; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.linkis.common.utils.Utils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.annotation.PostConstruct; -import java.util.Date; -import java.util.Map; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.TimeUnit; - - -@Component -public class PublishManager implements PublishListner { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private DSSProjectService projectService; - - private Map cacheMap = new ConcurrentHashMap<>(); - - @PostConstruct - public void init() { - logger.info("定时线程开启..."); - Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { - @Override - public void run() { - synchronized (cacheMap) { - cacheMap.entrySet().stream() - .filter(f -> (f.getValue().getStatus() == PublishJobEnum.Succeed || f.getValue().getStatus() == PublishJobEnum.Failed) && f.getValue().timeout()) - .forEach(f -> {logger.info("开始remove过期记录：{},更新时间{}",f.getKey(),f.getValue().getUpdateTime());cacheMap.remove(f.getKey());}); - } - } - }, 1, 5, TimeUnit.MINUTES); - Utils.defaultScheduler().scheduleAtFixedRate(new Runnable() { - @Override - public void run() { - synchronized (cacheMap) { - cacheMap.forEach((k, v) -> { - logger.info("cache中projectVersionID为：{}状态为：{}更新时间为：{}",k,v.getStatus(),v.getUpdateTime()); - }); - } - } - }, 1, 10, TimeUnit.MINUTES); - } - - @Override - public void onPublishSucceed(Long projectVersionID, String infoMsg) { - // TODO: 2019/6/21 目前不用加锁 - logger.info(infoMsg); - Date updateTime = new Date(); - PublishProjectCache cache = cacheMap.get(projectVersionID); - cache.setUpdateTime(updateTime); - cache.setStatus(PublishJobEnum.Succeed); - projectService.updatePublishHistory(projectVersionID, PublishJobEnum.Succeed.ordinal(), updateTime); - } - - @Override - public void onPublishFailed(Long projectVersionID, String errorMsg) { - logger.info(errorMsg); - Date updateTime = new Date(); - PublishProjectCache cache = cacheMap.get(projectVersionID); - cache.setUpdateTime(updateTime); - cache.setStatus(PublishJobEnum.Failed); - cache.setMsg(errorMsg); - projectService.updatePublishHistory(projectVersionID, PublishJobEnum.Failed.ordinal(), updateTime); - } - - @Override - public void onPublishRunning(Long projectVersionID, String infoMsg) { - logger.info(infoMsg); - Date updateTime = new Date(); - PublishProjectCache cache = cacheMap.get(projectVersionID); - cache.setUpdateTime(updateTime); - cache.setStatus(PublishJobEnum.Running); - projectService.updatePublishHistory(projectVersionID, PublishJobEnum.Running.ordinal(), updateTime); - } - - public void addPublishCache(Long projectVersionID, Long creatorID, String comment) throws DSSErrorException { - if (cacheMap.get(projectVersionID) == null) { - synchronized (cacheMap) { - if (cacheMap.get(projectVersionID) == null) { - DSSProjectPublishHistory history = projectService.getPublishHistoryByID(projectVersionID); - if (history == null) { - logger.info("创建一个发布job" + projectVersionID); - projectService.createPublishHistory(comment, creatorID, projectVersionID); - } else if (PublishJobEnum.Succeed.ordinal() == history.getState()) { - throw new DSSErrorException(90008, "该工程版本已经发布成功，请不要重复发布"); - } else if (PublishJobEnum.Failed.ordinal() == history.getState()) { - logger.info("数据库中的工程发布状态为失败，运行重新发布" + projectVersionID); - projectService.updatePublishHistory(projectVersionID, PublishJobEnum.Inited.ordinal(), new Date()); - } - // TODO: 2019/6/21 数据库中running或则inited的 - PublishProjectCache newCache = new PublishProjectCache(); - newCache.setStatus(PublishJobEnum.Inited); - newCache.setCreateTime(new Date()); - newCache.setUpdateTime(new Date()); - newCache.setProjectVersionID(projectVersionID); - cacheMap.put(projectVersionID, newCache); - } else { - throw new DSSErrorException(90006, "该工程版本已经发布过，请不要重复发布"); - } - } - } else { - PublishProjectCache cache = cacheMap.get(projectVersionID); - switch (cache.getStatus()) { - case Inited: - case Running: - case Succeed: - logger.info("当前cache状态为：" + cache.getStatus().toString()); - throw new DSSErrorException(90006, "该工程版本已经发布过，请不要重复发布"); - case Failed: - synchronized (cache) { - if (cache.getStatus() == PublishJobEnum.Failed) { - logger.info("当前cache状态为：Failed，准备转换为Inited"); - cache.setStatus(PublishJobEnum.Inited); - cache.setMsg(null); - cache.setUpdateTime(new Date()); - } else { - throw new DSSErrorException(90006, "该工程版本已经发布过，请不要重复发布"); - } - } - break; - default: - logger.error("unsupport status"); - } - } - } - - public PublishProjectCache getPublishCache(Long projectVersionID) { - return cacheMap.get(projectVersionID); - } - - public void checkeIsPublishing(Long projectVersionID) throws DSSErrorException { - PublishProjectCache publishCache = getPublishCache(projectVersionID); - if(publishCache != null && publishCache.isPublishing()){ - logger.info("工程正在发布中,projectVersionID:{}，状态为{}",projectVersionID,publishCache.getStatus()); - throw new DSSErrorException(90014,"工程正在发布中，不允许执行当前操作"); - } - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishProjectCache.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishProjectCache.java deleted file mode 100644 index 3072d5822..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishProjectCache.java +++ /dev/null @@ -1,81 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -import com.webank.wedatasphere.dss.server.conf.DSSServerConf; - -import java.util.Date; - - -public class PublishProjectCache { - - private Long projectVersionID; - private PublishJobEnum status; - private Date createTime; - private Date updateTime; - private String msg; - - public Boolean timeout(){ - return System.currentTimeMillis() - updateTime.getTime() > (int) DSSServerConf.CACHE_TIMEOUT.getValue(); - } - - public Boolean isPublishing() { - return status != null && (status == PublishJobEnum.Inited || status == PublishJobEnum.Running); - } - - public String getMsg() { - return msg; - } - - public void setMsg(String msg) { - this.msg = msg; - } - - public PublishJobEnum getStatus() { - return status; - } - - public void setStatus(PublishJobEnum status) { - this.status = status; - } - - public Long getProjectVersionID() { - return projectVersionID; - } - - public void setProjectVersionID(Long projectVersionID) { - this.projectVersionID = projectVersionID; - } - - public Date getCreateTime() { - return createTime; - } - - public void setCreateTime(Date createTime) { - this.createTime = createTime; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishSubmitJob.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishSubmitJob.java deleted file mode 100644 index af04335f5..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishSubmitJob.java +++ /dev/null @@ -1,50 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -import com.webank.wedatasphere.dss.server.service.DSSProjectService; - - -public class PublishSubmitJob extends PublishJob { - - private DSSProjectService projectService; - - private String comment; - - public PublishSubmitJob(DSSProjectService projectService, String user, String comment, Long projectVersionID) { - this.projectService = projectService; - this.user = user; - this.comment = comment; - this.projectVersionID = projectVersionID; - } - - @Override - public void run() { - try { - publishListner.onPublishRunning(projectVersionID,"开始发布工程："+projectVersionID); - projectService.publish(projectVersionID, user, comment); - publishListner.onPublishSucceed(projectVersionID,"工程发布成功："+ projectVersionID); - } catch (Exception e) { - logger.info("发布失败，原因：",e); - Throwable cause = e.getCause(); - String errorMsg = cause == null?e.getMessage():cause.getMessage(); - publishListner.onPublishFailed(projectVersionID,"工程发布失败："+ errorMsg); - } - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishSubmitJobDeamon.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishSubmitJobDeamon.java deleted file mode 100644 index 1cd79736b..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishSubmitJobDeamon.java +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - -import java.util.concurrent.Future; - - -public class PublishSubmitJobDeamon extends PublishJob { - - private Future future; - - private int time; - - public PublishSubmitJobDeamon(Future future, int time) { - this.future = future; - this.time = time; - } - - @Override - public void run() { - logger.info("projectVersionID:{}的发布守护线程启动", projectVersionID); - try { - Thread.sleep(1000L * time); - } catch (InterruptedException e) { - logger.error("deamon被打断了",e); - Thread.currentThread().interrupt(); - } - logger.info("开始取消projectVersionID:{}的发布线程", projectVersionID); - future.cancel(true); - logger.info("成功取消projectVersionID:{}的发布线程", projectVersionID); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishThreadPool.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishThreadPool.java deleted file mode 100644 index 9cf143d6b..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/publish/PublishThreadPool.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.publish; - - - -import com.webank.wedatasphere.linkis.common.utils.Utils; - -import java.util.concurrent.ExecutorService; - - -public class PublishThreadPool { - - private static ExecutorService executorService = Utils.newFixedThreadPool(1000, "project-publish", false); - private static ExecutorService executorServiceDeamon = Utils.newFixedThreadPool(1000, "project-publish-deamon", true); - public static ExecutorService get(){ - return executorService; - } - public static ExecutorService getDeamon(){ - return executorServiceDeamon; - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/FlowRestfulApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/FlowRestfulApi.java deleted file mode 100644 index 102518263..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/FlowRestfulApi.java +++ /dev/null @@ -1,180 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.restful; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.server.service.DSSUserService; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlowVersion; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.server.operate.Op; -import com.webank.wedatasphere.dss.server.publish.PublishManager; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.codehaus.jackson.JsonNode; -import org.codehaus.jackson.map.ObjectMapper; -import org.codehaus.jackson.type.TypeReference; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; -import org.springframework.util.StringUtils; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.*; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.io.IOException; -import java.util.Arrays; -import java.util.Date; -import java.util.List; - - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class FlowRestfulApi { - - @Autowired - private DSSFlowService flowService; - @Autowired - private DSSUserService dssUserService; - @Autowired - private PublishManager publishManager; - - ObjectMapper mapper = new ObjectMapper(); - - @GET - @Path("/listAllFlowVersions") - public Response listAllVersions(@Context HttpServletRequest req, @QueryParam("id")Long flowID,@QueryParam("projectVersionID")Long projectVersionID) { - List versions = flowService.listAllFlowVersions(flowID,projectVersionID); - return Message.messageToResponse(Message.ok().data("versions",versions)); - } - - @POST - @Path("/addFlow") - public Response addFlow(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - //如果是子工作流，那么分类应该是和父类一起的？ - String userName = SecurityFilter.getLoginUsername(req); - // TODO: 2019/5/23 flowName工程名下唯一校验 - String name = json.get("name").getTextValue(); - String description = json.get("description") == null?null:json.get("description").getTextValue(); - Long parentFlowID = json.get("parentFlowID") ==null?null:json.get("parentFlowID").getLongValue(); - Long taxonomyID = json.get("taxonomyID") == null? null:json.get("taxonomyID").getLongValue(); - Long projectVersionID = json.get("projectVersionID").getLongValue(); - String uses = json.get("uses") == null?null:json.get("uses").getTextValue(); - if(taxonomyID == null && parentFlowID == null) throw new DSSErrorException(90009,"请求选择工作流分类"); - publishManager.checkeIsPublishing(projectVersionID); - DSSFlow dssFlow = new DSSFlow(); - dssFlow.setProjectID(projectService.getProjectByProjectVersionID(projectVersionID).getId()); - dssFlow.setName(name); - dssFlow.setDescription(description); - dssFlow.setCreatorID(dssUserService.getUserID(userName)); - dssFlow.setCreateTime(new Date()); - dssFlow.setState(false); - dssFlow.setSource("create by user"); - dssFlow.setUses(uses); - if(parentFlowID == null){ - dssFlow.setRootFlow(true); - dssFlow.setRank(0); - dssFlow.setHasSaved(true); - dssFlow = flowService.addRootFlow(dssFlow,taxonomyID,projectVersionID); - }else { - dssFlow.setRootFlow(false); - Integer rank = flowService.getParentRank(parentFlowID); - // TODO: 2019/6/3 并发问题考虑for update - dssFlow.setRank(rank+1); - dssFlow.setHasSaved(false); - dssFlow = flowService.addSubFlow(dssFlow,parentFlowID,projectVersionID); - } - // TODO: 2019/5/16 空值校验，重复名校验 - return Message.messageToResponse(Message.ok().data("flow", dssFlow)); - } - - @POST - @Path("/updateFlowBaseInfo") - public Response updateFlowBaseInfo(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - Long flowID = json.get("id").getLongValue(); - String name = json.get("name")==null?null:json.get("name").getTextValue(); - String description = json.get("description") == null?null:json.get("description").getTextValue(); - Long taxonomyID = json.get("taxonomyID") == null?null:json.get("taxonomyID").getLongValue(); - Long projectVersionID = json.get("projectVersionID").getLongValue(); - String uses = json.get("uses") == null?null:json.get("uses").getTextValue(); - publishManager.checkeIsPublishing(projectVersionID); - // TODO: 2019/6/13 projectVersionID的更新校验 - //这里可以不做事务 - DSSFlow dssFlow = new DSSFlow(); - dssFlow.setId(flowID); - dssFlow.setName(name); - dssFlow.setDescription(description); - dssFlow.setUses(uses); - flowService.updateFlowBaseInfo(dssFlow,projectVersionID,taxonomyID); - return Message.messageToResponse(Message.ok()); - } - - @GET - @Path("/get") - public Response get(@Context HttpServletRequest req, @QueryParam("id")Long flowID,@QueryParam("version")String version,@QueryParam("projectVersionID")Long projectVersionID) throws DSSErrorException { - // TODO: 2019/5/23 id空值判断 - DSSFlow dssFlow; - if (StringUtils.isEmpty(version)){ - dssFlow = flowService.getLatestVersionFlow(flowID,projectVersionID); - dssFlow.setFlowVersions(Arrays.asList(dssFlow.getLatestVersion())); - }else { - dssFlow = flowService.getOneVersionFlow(flowID, version,projectVersionID); - } - return Message.messageToResponse(Message.ok().data("flow", dssFlow)); - } - - @POST - @Path("/deleteFlow") - public Response deleteFlow(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - Long flowID = json.get("id").getLongValue(); - Boolean sure = json.get("sure") == null?false:json.get("sure").getBooleanValue(); - Long projectVersionID = json.get("projectVersionID").getLongValue(); - // TODO: 2019/6/13 projectVersionID的更新校验 - //state为true代表曾经发布过 - if(flowService.getFlowByID(flowID).getState() && !sure) { - return Message.messageToResponse(Message.ok().data("warmMsg","该工作流曾经发布过，删除将会将该工作流的所有版本都删除，是否继续？")); - } - publishManager.checkeIsPublishing(projectVersionID); - flowService.batchDeleteFlow(Arrays.asList(flowID),projectVersionID); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/saveFlow") - public Response saveFlow(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException, IOException, AppJointErrorException { - Long flowID = json.get("id").getLongValue(); - String jsonFlow = json.get("json").getTextValue(); - Long projectVersionID = json.get("projectVersionID").getLongValue(); - String userName = SecurityFilter.getLoginUsername(req); - String comment = json.get("comment") == null?"保存更新":json.get("comment").getTextValue(); - List ops = mapper.readValue(json.get("ops"), new TypeReference>(){}); - publishManager.checkeIsPublishing(projectVersionID); - String version =flowService.saveFlow(flowID,jsonFlow,comment,userName,projectVersionID,ops); - return Message.messageToResponse(Message.ok().data("flowVersion",version)); - } - - @Autowired - private DSSProjectService projectService; - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/FlowTaxonomyRestfulApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/FlowTaxonomyRestfulApi.java deleted file mode 100644 index bc43c5ab6..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/FlowTaxonomyRestfulApi.java +++ /dev/null @@ -1,113 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.restful; - - -import com.webank.wedatasphere.dss.server.service.DSSFlowTaxonomyService; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.server.service.DSSUserService; -import com.webank.wedatasphere.dss.server.entity.DSSFlowTaxonomy; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.server.publish.PublishManager; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.codehaus.jackson.JsonNode; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.Consumes; -import javax.ws.rs.POST; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.util.Date; - - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class FlowTaxonomyRestfulApi { - - @Autowired - private DSSFlowTaxonomyService flowTaxonomyService; - @Autowired - private DSSUserService dssUserService; - @Autowired - private DSSProjectService projectService; - @Autowired - private PublishManager publishManager; - - @POST - @Path("/addFlowTaxonomy") - public Response addFlowTaxonomy(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - String userName = SecurityFilter.getLoginUsername(req); - String name = json.get("name").getTextValue(); - String description = json.get("description").getTextValue(); - Long projectVersionID = json.get("projectVersionID").getLongValue(); - publishManager.checkeIsPublishing(projectVersionID); - // TODO: 2019/5/16 空值校验，重复名校验 - DSSFlowTaxonomy dssFlowTaxonomy = new DSSFlowTaxonomy(); - Date date = new Date(); - dssFlowTaxonomy.setName(name); - dssFlowTaxonomy.setDescription(description); - dssFlowTaxonomy.setCreatorID(dssUserService.getUserID(userName)); - dssFlowTaxonomy.setCreateTime(date); - dssFlowTaxonomy.setUpdateTime(date); - dssFlowTaxonomy.setProjectID(projectService.getProjectByProjectVersionID(projectVersionID).getId()); - flowTaxonomyService.addFlowTaxonomy(dssFlowTaxonomy,projectVersionID); - return Message.messageToResponse(Message.ok()); - } - @POST - @Path("/updateFlowTaxonomy") - public Response updateFlowTaxonomy(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - String name = json.get("name")==null?null:json.get("name").getTextValue(); - String description = json.get("description") == null?null:json.get("description").getTextValue(); - Long id = json.get("id").getLongValue(); - Long projectVersionID = json.get("projectVersionID").getLongValue(); - publishManager.checkeIsPublishing(projectVersionID); - // TODO: 2019/6/13 projectVersionID的更新校验 - // TODO: 2019/5/16 空值校验，重复名校验 - DSSFlowTaxonomy dssFlowTaxonomy = new DSSFlowTaxonomy(); - dssFlowTaxonomy.setId(id); - dssFlowTaxonomy.setName(name); - dssFlowTaxonomy.setDescription(description); - dssFlowTaxonomy.setUpdateTime(new Date()); - flowTaxonomyService.updateFlowTaxonomy(dssFlowTaxonomy,projectVersionID); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/deleteFlowTaxonomy") - public Response deleteFlowTaxonomy(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - Long id = json.get("id").getLongValue(); - // TODO: 2019/5/16 工程分类下工程是不是空的校验 - // TODO: 2019/5/16 如果删除的是我的工程，我参与的工程 - Long projectVersionID = json.get("projectVersionID").getLongValue(); - publishManager.checkeIsPublishing(projectVersionID); - // TODO: 2019/6/13 projectVersionID的更新校验 - if(flowTaxonomyService.hasFlows(id)){ - throw new DSSErrorException(90001,"该工作流分类下还有工作流，不能删除该分类！"); - } - flowTaxonomyService.deleteFlowTaxonomy(id,projectVersionID); - return Message.messageToResponse(Message.ok()); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/NodeRestfulApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/NodeRestfulApi.java deleted file mode 100644 index ec1c43a22..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/NodeRestfulApi.java +++ /dev/null @@ -1,115 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.restful; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.core.CommonAppJointNode; -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import com.webank.wedatasphere.dss.application.util.ApplicationUtils; -import com.webank.wedatasphere.dss.server.entity.NodeInfo; -import com.webank.wedatasphere.dss.server.function.FunctionInvoker; -import com.webank.wedatasphere.dss.server.function.FunctionPool; -import com.webank.wedatasphere.dss.server.function.NodeServiceFunction; -import com.webank.wedatasphere.dss.server.service.DSSNodeInfoService; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.*; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class NodeRestfulApi { - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private DSSNodeInfoService nodeInfoService; - - @Autowired - private ApplicationService applicationService; - - @Autowired - private FunctionInvoker functionInvoker; - - @GET - @Path("/listNodeType") - public Response listNodeType(@Context HttpServletRequest req, @QueryParam("applicationName")List names) { - HashMap> nodeTypes = new HashMap<>(); - if(names == null || names.isEmpty()){ - names = applicationService.listApplicationNames(); - } - names.forEach(f ->{ - //如果redirecturl不为ull的话，返回固定格式，并且进行url加密 - Application application = applicationService.getApplication(f); - List nodeInfo = nodeInfoService.getNodeType(f); - String redirectUrl = application.getRedirectUrl(); - if(redirectUrl != null){ - nodeInfo.forEach(n ->n.setJumpUrl(ApplicationUtils.redirectUrlFormat(redirectUrl,n.getJumpUrl()))); - } - nodeTypes.put(f,nodeInfo); - }); - return Message.messageToResponse(Message.ok().data("nodeTypes",nodeTypes)); - } - - @POST - @Path("/createAppjointNode") - public Response createExternalNode(@Context HttpServletRequest req,Map json) throws AppJointErrorException { - return createNodeResponse(req,json, FunctionPool.createNode); - } - - private Response createNodeResponse(HttpServletRequest req, Map json, NodeServiceFunction function) throws AppJointErrorException { - String userName = SecurityFilter.getLoginUsername(req); - Long projectID = Long.parseLong(json.get("id").toString()); - String nodeType = json.get("nodeType").toString(); - Map params = (Map)json.get("params"); - logger.info("CreateExternalNode request params is "+ params+",nodeType:"+nodeType); - CommonAppJointNode node = new CommonAppJointNode(); - node.setProjectId(projectID); - node.setNodeType(nodeType); - Map jobContent = functionInvoker.nodeServiceFunction(userName,params,node,function); - return Message.messageToResponse(Message.ok().data("result",jobContent)); - } - - @POST - @Path("/updateAppjointNode") - public Response updateExternalNode(@Context HttpServletRequest req,Map json) throws AppJointErrorException { - return createNodeResponse(req,json, FunctionPool.updateNode); - } - - @POST - @Path("/deleteAppjointNode") - public Response deleteExternalNode(@Context HttpServletRequest req,Map json) throws AppJointErrorException { - return createNodeResponse(req,json, FunctionPool.deleteNode); - } -} - diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/ProjectRestfulApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/ProjectRestfulApi.java deleted file mode 100644 index be22ec758..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/ProjectRestfulApi.java +++ /dev/null @@ -1,201 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.restful; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectVersion; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.server.dao.DSSUserMapper; -import com.webank.wedatasphere.dss.server.dao.ProjectMapper; -import com.webank.wedatasphere.dss.server.entity.ApplicationArea; -import com.webank.wedatasphere.dss.server.publish.*; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.server.service.DSSProjectTaxonomyService; -import com.webank.wedatasphere.dss.server.service.DSSUserService; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.codehaus.jackson.JsonNode; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.*; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; -import java.util.concurrent.Future; - - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class ProjectRestfulApi { - - @Autowired - private DSSProjectTaxonomyService projectTaxonomyService; - @Autowired - private DSSProjectService projectService; - @Autowired - private DSSUserService dssUserService; - @Autowired - private ProjectMapper projectMapper; - @Autowired - private PublishJobFactory publishJobFactory; - @Autowired - private PublishManager publishManager; - - @Autowired - private DSSUserMapper dssUserMapper; - - @GET - @Path("/listAllProjectVersions") - public Response listAllVersions(@Context HttpServletRequest req, @QueryParam("id") Long projectID) { - List versions = projectService.listAllProjectVersions(projectID); - return Message.messageToResponse(Message.ok().data("versions", versions)); - } - - @GET - @Path("/listApplicationAreas") - public Response listApplicationAreas(@Context HttpServletRequest req) { - String header = req.getHeader("Content-language").trim(); - ApplicationArea[] applicationAreas = ApplicationArea.values(); - List areas = new ArrayList<>(); - Arrays.stream(applicationAreas).forEach(item ->{ - if ("zh-CN".equals(header)){ - areas.add( item.getName()); - }else { - areas.add(item.getEnName()); - } - }); - return Message.messageToResponse(Message.ok().data("applicationAreas", areas)); - } - - @POST - @Path("/addProject") - public Response addProject(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException,AppJointErrorException { - String userName = SecurityFilter.getLoginUsername(req); - String name = json.get("name").getTextValue(); - String description = json.get("description").getTextValue(); - Long taxonomyID = json.get("taxonomyID") == null? null:json.get("taxonomyID").getLongValue(); - String product = json.get("product").getTextValue(); - Integer applicationArea = json.get("applicationArea").getIntValue(); - String business = json.get("business").getTextValue(); - Long workspaceId = json.get("workspaceId") == null ? 1 : json.get("workspaceId").getLongValue(); - // TODO: 2019/5/16 空值校验，重复名校验 - projectService.addProject(userName, name, description, taxonomyID,product,applicationArea,business, workspaceId); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/updateProject") - public Response updateProject(@Context HttpServletRequest req, JsonNode json) throws AppJointErrorException { - String userName = SecurityFilter.getLoginUsername(req); - String name = json.get("name") == null ? null : json.get("name").getTextValue(); - String description = json.get("description") == null ? null : json.get("description").getTextValue(); - long projectID = json.get("id").getLongValue(); - Long taxonomyID = json.get("taxonomyID") == null ? null : json.get("taxonomyID").getLongValue(); - String product = json.get("product").getTextValue(); - Integer applicationArea = json.get("applicationArea").getIntValue(); - String business = json.get("business").getTextValue(); - //这里可以不做事务 - projectService.updateProject(projectID, name, description, userName , product ,applicationArea ,business); - projectTaxonomyService.updateProjectTaxonomyRelation(projectID, taxonomyID); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/deleteProject") - public Response deleteProject(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - String userName = SecurityFilter.getLoginUsername(req); - Long projectID = json.get("id").getLongValue(); - Boolean sure = json.get("sure") != null&&json.get("sure").getBooleanValue(); - Boolean ifDelScheduler = json.get("ifDelScheduler") != null &&json.get("ifDelScheduler").getBooleanValue(); - if (projectService.isPublished(projectID) && !sure) { - return Message.messageToResponse(Message.ok().data("warmMsg", "该工程发布过，是否继续？")); - } - // TODO: 2019/5/23 阻止projectID为-1 - projectService.deleteProject(projectID, userName,ifDelScheduler); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/copyProjectAndFlow") - public Response copyProject(@Context HttpServletRequest req, JsonNode json) throws InterruptedException, DSSErrorException, AppJointErrorException { - String userName = SecurityFilter.getLoginUsername(req); - Long projectID = json.get("projectID").getLongValue(); - String projectName = json.get("projectName") == null ? null : json.get("projectName").getTextValue(); - DSSProjectVersion maxVersion = projectMapper.selectLatestVersionByProjectID(projectID); - projectService.copyProject( maxVersion.getId(),projectID, projectName, userName); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/copyVersionAndFlow") - public Response copyProjectVersion(@Context HttpServletRequest req, JsonNode json) throws InterruptedException, DSSErrorException { - String userName = SecurityFilter.getLoginUsername(req); - Long copyprojectVersionID = json.get("projectVersionID").getLongValue(); - DSSProjectVersion currentVersion = projectMapper.selectProjectVersionByID(copyprojectVersionID); - DSSProjectVersion maxVersion = projectMapper.selectLatestVersionByProjectID(currentVersion.getProjectID()); - - projectService.copyProjectVersionMax( maxVersion.getId(), maxVersion,currentVersion,userName,null); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/publish") - public Response publish(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - String userName = SecurityFilter.getLoginUsername(req); - Long projectID = json.get("id").getLongValue(); - String comment = json.get("comment").getTextValue(); - DSSProject latestVersionProject = projectService.getLatestVersionProject(projectID); - publishManager.addPublishCache(latestVersionProject.getLatestVersion().getId(), dssUserService.getUserID(userName),comment); - PublishSubmitJob job = publishJobFactory.createSubmitPublishJob(latestVersionProject.getLatestVersion().getId(), userName, comment); - Future submit = PublishThreadPool.get().submit(job); - PublishSubmitJobDeamon deamon = publishJobFactory.createSubmitPublishJobDeamon(submit, job); - PublishThreadPool.getDeamon().execute(deamon); - return Message.messageToResponse(Message.ok()); - } - - @GET - @Path("/publishQuery") - public Response publishQuery(@Context HttpServletRequest req, @QueryParam("projectVersionID") Long projectVersionID) { - PublishProjectCache cache = publishManager.getPublishCache(projectVersionID); - return Message.messageToResponse(Message.ok().data("info",cache)); - } - - @GET - @Path("/getAppJointProjectID") - public Response getAppjointProjectID(@Context HttpServletRequest req, @QueryParam("projectID") Long projectID,@QueryParam("nodeType") String nodeType) { - Long appJointProjectID = projectService.getAppjointProjectID(projectID,nodeType); - return Message.messageToResponse(Message.ok().data("appJointProjectID",appJointProjectID)); - } - - @GET - @Path("/getAppjointProjectIDByApplicationName") - public Response listAppJointProjectID(@Context HttpServletRequest req, @QueryParam("projectID") Long projectID,@QueryParam("applicationName") String applicationName) { - Long appJointProjectID = projectService.getAppjointProjectIDByApplicationName(projectID,applicationName); - return Message.messageToResponse(Message.ok().data("appJointProjectID",appJointProjectID)); - } - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/ProjectTaxonomyRestfulApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/ProjectTaxonomyRestfulApi.java deleted file mode 100644 index 4852c0fe9..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/ProjectTaxonomyRestfulApi.java +++ /dev/null @@ -1,98 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.restful; - - -import com.webank.wedatasphere.dss.server.service.DSSProjectTaxonomyService; -import com.webank.wedatasphere.dss.server.service.DSSUserService; -import com.webank.wedatasphere.dss.server.entity.DSSProjectTaxonomy; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.codehaus.jackson.JsonNode; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.Consumes; -import javax.ws.rs.POST; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.util.Date; - - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class ProjectTaxonomyRestfulApi { - - @Autowired - private DSSProjectTaxonomyService projectTaxonomyService; - @Autowired - private DSSUserService dssUserService; - - @POST - @Path("/addProjectTaxonomy") - public Response addProjectTaxonomy(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - String userName = SecurityFilter.getLoginUsername(req); - String name = json.get("name").getTextValue(); - String description = json.get("description").getTextValue(); - // TODO: 2019/5/16 空值校验，重复名校验 - DSSProjectTaxonomy dssProjectTaxonomy = new DSSProjectTaxonomy(); - Date date = new Date(); - dssProjectTaxonomy.setName(name); - dssProjectTaxonomy.setDescription(description); - dssProjectTaxonomy.setCreatorID(dssUserService.getUserID(userName)); - dssProjectTaxonomy.setCreateTime(date); - dssProjectTaxonomy.setUpdateTime(date); - projectTaxonomyService.addProjectTaxonomy(dssProjectTaxonomy); - return Message.messageToResponse(Message.ok()); - } - @POST - @Path("/updateProjectTaxonomy") - public Response updateProjectTaxonomy(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - String name = json.get("name")==null?null:json.get("name").getTextValue(); - String description = json.get("description") == null?null:json.get("description").getTextValue(); - Long id = json.get("id").getLongValue(); - // TODO: 2019/5/16 空值校验，重复名校验 - DSSProjectTaxonomy dssProjectTaxonomy = new DSSProjectTaxonomy(); - dssProjectTaxonomy.setId(id); - dssProjectTaxonomy.setName(name); - dssProjectTaxonomy.setDescription(description); - dssProjectTaxonomy.setUpdateTime(new Date()); - projectTaxonomyService.updateProjectTaxonomy(dssProjectTaxonomy); - return Message.messageToResponse(Message.ok()); - } - - @POST - @Path("/deleteProjectTaxonomy") - public Response deleteProjectTaxonomy(@Context HttpServletRequest req, JsonNode json) throws DSSErrorException { - Long id = json.get("id").getLongValue(); - // TODO: 2019/5/16 工程分类下工程是不是空的校验 - // TODO: 2019/5/16 如果删除的是我的工程，我参与的工程 - if(projectTaxonomyService.hasProjects(id)){ - throw new DSSErrorException(60000,"该工程分类下还有工程，不能删除该分类！"); - } - projectTaxonomyService.deleteProjectTaxonomy(id); - return Message.messageToResponse(Message.ok()); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/TreeRestfulApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/TreeRestfulApi.java deleted file mode 100644 index fde36296e..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/TreeRestfulApi.java +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.restful; - - - -import com.webank.wedatasphere.dss.server.crumb.CrumbFactory; -import com.webank.wedatasphere.dss.server.crumb.QuerParamsParser; -import com.webank.wedatasphere.dss.server.entity.Crumb; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; -import org.springframework.web.bind.annotation.RequestParam; -import scala.Enumeration; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.*; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; - - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class TreeRestfulApi { - @Autowired - private CrumbFactory crumbFactory; - - @GET - @Path("/tree") - public Response Crumb(@Context HttpServletRequest req) { - String userName = SecurityFilter.getLoginUsername(req); - String queryString = req.getQueryString() == null ? "" : req.getQueryString(); - String[] params = new String[]{} ; - if (!"".equals(queryString)) params = QuerParamsParser.toArray(queryString); - Enumeration.Value crumbType = QuerParamsParser.getCrumbType(queryString); - Crumb[] crumbs = crumbFactory.createCrumbs(crumbType, params); - Object crumbData = crumbFactory.createCrumbData(crumbType, QuerParamsParser.toMap(params),userName); - return Message.messageToResponse(Message.ok().data("crumbs", crumbs).data("data",crumbData)); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/UserManagerApi.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/UserManagerApi.java deleted file mode 100644 index a249e7ae3..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/restful/UserManagerApi.java +++ /dev/null @@ -1,92 +0,0 @@ -package com.webank.wedatasphere.dss.server.restful; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.scheduler.SchedulerAppJoint; -import com.webank.wedatasphere.dss.application.conf.ApplicationConf; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import com.webank.wedatasphere.dss.server.constant.DSSServerConstant; -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphere.linkis.server.security.SecurityFilter; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.impl.UserAuthorizationClient; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.ws.rs.*; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; - -/** - * @program: user-authorization - * @description: 鲁班对外交互的接口 包括施工 注册用户 - * @create: 2020-08-12 14:24 - **/ - -@Component -@Path("/dss") -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class UserManagerApi { - - private UserAuthorizationClient client = new UserAuthorizationClient(); - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private ApplicationService applicationService; - - private SchedulerAppJoint schedulerAppJoint; - - @POST - @Path("/user") - public Response createUser(@Context HttpServletRequest req, AuthorizationBody body) { - String username = SecurityFilter.getLoginUsername(req); - String superUserName = ApplicationConf.SUPER_USER_NAME; - if(!username.equals(superUserName)){ - return Message.messageToResponse(Message.error(DSSServerConstant.SUPER_USER_LOGIN_ERROR)); - } - - try { - String result = client.authorization(body); - - if(result.equals(AbsCommand.SUCCESS)){ - schedulerAppJoint = getSchedulerAppJoint(); - if(schedulerAppJoint != null){ - try{ - schedulerAppJoint.getSecurityService().reloadToken(); - }catch (Throwable throwable){ - logger.warn("choose schedulies,don not care"); - } - - } - return Message.messageToResponse(Message.ok()); - }else { - return Message.messageToResponse(Message.error(AbsCommand.ERROR)); - } - } catch (Exception e) { - return Message.messageToResponse(Message.error(e.getMessage())); - } - - - - - } - - private SchedulerAppJoint getSchedulerAppJoint(){ - - if(schedulerAppJoint == null){ - try { - schedulerAppJoint = (SchedulerAppJoint)applicationService.getAppjoint("schedulis"); - } catch (AppJointErrorException e) { - logger.error("Schedule system init failed!", e); - } - } - return schedulerAppJoint; - } - - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSFlowService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSFlowService.java deleted file mode 100644 index 12d0f5770..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSFlowService.java +++ /dev/null @@ -1,74 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlowVersion; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.server.operate.Op; - -import java.util.List; - - -public interface DSSFlowService { - DSSFlow getFlowByID(Long id); - - List listAllFlowVersions(Long flowID, Long projectVersionID); - - DSSFlow addRootFlow(DSSFlow dssFlow, Long taxonomyID, Long projectVersionID) throws DSSErrorException; - - DSSFlow addSubFlow(DSSFlow dssFlow, Long parentFlowID, Long projectVersionID) throws DSSErrorException; - - /** - * 通过flowID获取最新版本的DSSFlow，版本信息在latestVersion - * @param flowID - * @return - */ - DSSFlow getLatestVersionFlow(Long flowID, Long projectVersionID) throws DSSErrorException; - - /** - * 通过flowID和某个版本号，获取一个DSSFlow，版本信息在versions数组中的第一个元素 - * @param flowID - * @return - */ - DSSFlow getOneVersionFlow(Long flowID, String version, Long projectVersionID); - - /** - * 通过DSSFlow对象拿到最新的json，其实这里只要个flowID应该就可以了 - * @param dssFlow - * @return - */ -/* String getLatestJsonByFlow(dssFlow dssFlow); - - dssFlow getLatestVersionFlow(Long ProjectID,String flowName);*/ - - void updateFlowBaseInfo(DSSFlow dssFlow, Long projectVersionID, Long taxonomyID) throws DSSErrorException; - - void updateFlowTaxonomyRelation(Long flowID, Long taxonomyID) throws DSSErrorException; - - void batchDeleteFlow(List flowIDlist, Long projectVersionID); - - String saveFlow(Long flowID, String jsonFlow, String comment, String userName, Long projectVersionID, List ops) throws DSSErrorException, AppJointErrorException; - - Integer getParentRank(Long flowID); - - DSSFlowVersion getLatestVersionByFlowIDAndProjectVersionID(Long flowID, Long projectVersionID); - - Long getParentFlowID(Long id); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSFlowTaxonomyService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSFlowTaxonomyService.java deleted file mode 100644 index f105779a3..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSFlowTaxonomyService.java +++ /dev/null @@ -1,41 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - - -import com.webank.wedatasphere.dss.server.entity.DSSFlowTaxonomy; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - -import java.util.List; - - -public interface DSSFlowTaxonomyService { - DSSFlowTaxonomy getFlowTaxonomyByID(Long id); - - //--------- - List listAllFlowTaxonomy(Long projectVersionID, Boolean isRootFlow); - List listFlowTaxonomy(Long projectVersionID, Long flowTaxonomyID, Boolean isRootFlow); - - void addFlowTaxonomy(DSSFlowTaxonomy dssFlowTaxonomy, Long projectVersionID) throws DSSErrorException; - - void updateFlowTaxonomy(DSSFlowTaxonomy dssFlowTaxonomy, Long projectVersionID) throws DSSErrorException; - - boolean hasFlows(Long flowTaxonomyID); - - void deleteFlowTaxonomy(Long flowTaxonomyID, Long projectVersionID); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSNodeInfoService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSNodeInfoService.java deleted file mode 100644 index dc7303f52..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSNodeInfoService.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - - - - -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.server.entity.NodeInfo; - -import java.util.List; - - -public interface DSSNodeInfoService { - - List getNodeType(String f); - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSProjectService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSProjectService.java deleted file mode 100644 index 1cad4fd68..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSProjectService.java +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectPublishHistory; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectVersion; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.common.protocol.RequestDSSProject; - -import java.util.Date; -import java.util.List; - - -public interface DSSProjectService { - - DSSProject getProjectByID(Long id); - - Long addProject(String userName, String name, String description, Long taxonomyID,String product,Integer applicationArea,String business, Long workspaceId) throws DSSErrorException, AppJointErrorException; - - void updateProject(long projectID, String name, String description, String userName , String product ,Integer applicationArea ,String business) throws AppJointErrorException; - - void deleteProject(long projectID, String userName, Boolean ifDelScheduler) throws DSSErrorException; - - DSSProject getLatestVersionProject(Long projectID); - - DSSProject getProjectByProjectVersionID(Long projectVersionID); - - Boolean isPublished(Long projectID); - - List listAllProjectVersions(Long projectID); - - Long copyProject(Long projectVersionID, Long projectID, String projectName, String userName) throws DSSErrorException, InterruptedException, AppJointErrorException; - - void copyProjectVersionMax(Long projectVersionID, DSSProjectVersion maxVersion, DSSProjectVersion copyVersion, String userName, Long WTSSprojectID) throws DSSErrorException, InterruptedException; - - void publish(Long projectVersionID, String userName, String comment) throws DSSErrorException, InterruptedException, AppJointErrorException; - - Long createPublishHistory(String comment, Long creatorID, Long projectVersionID) throws DSSErrorException; - - void updatePublishHistory(Long projectVersionID, Integer status, Date updateTime); - - DSSProjectPublishHistory getPublishHistoryByID(Long projectVersionID); - - DSSProject getExecutionDSSProject(RequestDSSProject requestDSSProject) throws DSSErrorException; - - Long getAppjointProjectID(Long projectID, String nodeType); - - Long getAppjointProjectIDByApplicationName(Long projectID, String applicationName); -} - diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSProjectTaxonomyService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSProjectTaxonomyService.java deleted file mode 100644 index cebabbf63..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSProjectTaxonomyService.java +++ /dev/null @@ -1,46 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - -import com.webank.wedatasphere.dss.server.entity.DSSProjectTaxonomy; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; - -import java.util.List; - - -public interface DSSProjectTaxonomyService { - - DSSProjectTaxonomy getProjectTaxonomyByID(Long id); - - List listProjectTaxonomyByUser(String userName); - - //---------------------- - List listAllProjectTaxonomy(String userName, Long workspaceId); - - List listProjectTaxonomy(Long taxonomyID, String userName); - - void addProjectTaxonomy(DSSProjectTaxonomy dssProjectTaxonomy) throws DSSErrorException; - - void updateProjectTaxonomy(DSSProjectTaxonomy dssProjectTaxonomy) throws DSSErrorException; - - boolean hasProjects(Long projectTaxonomyID); - - void deleteProjectTaxonomy(Long projectTaxonomyID); - - void updateProjectTaxonomyRelation(Long projectID, Long taxonomyID); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSUserService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSUserService.java deleted file mode 100644 index 2c2760297..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSUserService.java +++ /dev/null @@ -1,23 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - - -public interface DSSUserService { - Long getUserID(String userName); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSWorkspaceService.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSWorkspaceService.java deleted file mode 100644 index f88c221ea..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/DSSWorkspaceService.java +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service; - -import com.webank.wedatasphere.dss.server.dto.response.HomepageDemoMenuVo; -import com.webank.wedatasphere.dss.server.dto.response.HomepageVideoVo; -import com.webank.wedatasphere.dss.server.dto.response.OnestopMenuVo; -import com.webank.wedatasphere.dss.server.entity.DSSWorkspace; -import com.webank.wedatasphere.dss.server.dto.response.WorkspaceDepartmentVo; -import com.webank.wedatasphere.dss.server.dto.response.*; - -import java.util.List; - -/** - * Created by schumiyi on 2020/6/22 - */ -public interface DSSWorkspaceService { - List getWorkspaces(); - - Long addWorkspace(String userName, String name, String department, String label, String description); - - boolean existWorkspaceName(String name); - - List getWorkSpaceDepartments(); - - List getHomepageDemos(boolean isChinese); - - List getHomepageVideos(boolean isChinese); - - List getWorkspaceManagements(Long workspaceId, String username, boolean isChinese); - - List getWorkspaceApplications(Long workspaceId, String username, boolean isChinese); - - DSSWorkspace getWorkspacesById(Long id); - - /** - * 查询用户收藏的应用，如果是新用户，就在数据库给它插入默认两个收藏：脚本开发与工作流 workflow scriptis - * @param workspaceId - * @param username - * @param isChinese - * @return - */ - List getWorkspaceFavorites(Long workspaceId, String username, boolean isChinese); - - Long addFavorite(String username, Long workspaceId, Long menuApplicationId); - - Long deleteFavorite(String username, Long favouritesId); -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSFlowServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSFlowServiceImpl.java deleted file mode 100644 index 218aacc44..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSFlowServiceImpl.java +++ /dev/null @@ -1,282 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlowVersion; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.server.dao.DSSUserMapper; -import com.webank.wedatasphere.dss.server.dao.FlowMapper; -import com.webank.wedatasphere.dss.server.dao.FlowTaxonomyMapper; -import com.webank.wedatasphere.dss.server.lock.Lock; -import com.webank.wedatasphere.dss.server.operate.Op; -import com.webank.wedatasphere.dss.server.operate.Operate; -import com.webank.wedatasphere.dss.server.service.BMLService; -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.dao.DuplicateKeyException; -import org.springframework.stereotype.Service; -import org.springframework.transaction.annotation.Transactional; - -import java.util.*; -import java.util.stream.Collectors; - - -@Service -public class DSSFlowServiceImpl implements DSSFlowService { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private FlowMapper flowMapper; - @Autowired - private FlowTaxonomyMapper flowTaxonomyMapper; - @Autowired - private DSSUserMapper dssUserMapper; - - @Autowired - private DSSProjectService projectService; - @Autowired - private Operate[] operates; - @Autowired - private BMLService bmlService; - - @Override - public DSSFlow getFlowByID(Long id) { - return flowMapper.selectFlowByID(id); - } - - @Override - public List listAllFlowVersions(Long flowID, Long projectVersionID) { - List versions = flowMapper.listFlowVersionsByFlowID(flowID, projectVersionID).stream().sorted((v1, v2) -> { - return v1.compareTo(v2); - }).collect(Collectors.toList()); - return versions; - } - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public DSSFlow addRootFlow(DSSFlow dssFlow, Long taxonomyID, Long projectVersionID) throws DSSErrorException { - try { - flowMapper.insertFlow(dssFlow); - } catch (DuplicateKeyException e) { - logger.info(e.getMessage()); - throw new DSSErrorException(90003, "工作流名不能重复"); - } - //通过resource上传空文件，获取resourceId(jsonPath)和version - Map bmlReturnMap = bmlService.upload(dssUserMapper.getuserName(dssFlow.getCreatorID()), "", UUID.randomUUID().toString() + ".json"); - //数据库中插入版本信息 - DSSFlowVersion version = new DSSFlowVersion(); - version.setFlowID(dssFlow.getId()); - version.setSource("create by user"); - version.setJsonPath(bmlReturnMap.get("resourceId").toString()); - version.setVersion(bmlReturnMap.get("version").toString()); - version.setUpdateTime(new Date()); - version.setUpdatorID(dssFlow.getCreatorID()); - // TODO: 2019/6/12 这里应该由前台传入 - version.setProjectVersionID(projectVersionID); - flowMapper.insertFlowVersion(version); - //数据库中插入分类关联信息 - flowTaxonomyMapper.insertFlowTaxonomyRelation(taxonomyID, dssFlow.getId()); - return dssFlow; - } - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public DSSFlow addSubFlow(DSSFlow dssFlow, Long parentFlowID, Long projectVersionID) throws DSSErrorException { - Long taxonomyID = flowTaxonomyMapper.selectTaxonomyIDByFlowID(parentFlowID); - DSSFlow parentFlow = flowMapper.selectFlowByID(parentFlowID); - dssFlow.setProjectID(parentFlow.getProjectID()); - DSSFlow subFlow = addRootFlow(dssFlow, taxonomyID, projectVersionID); - //数据库中插入关联信息 - flowMapper.insertFlowRelation(subFlow.getId(), parentFlowID); - return subFlow; - } - - @Override - public DSSFlow getLatestVersionFlow(Long flowID, Long projectVersionID) throws DSSErrorException { - DSSFlow dssFlow = getFlowByID(flowID); - DSSFlowVersion dssFlowVersion = getLatestVersionByFlowIDAndProjectVersionID(flowID, projectVersionID); - if (dssFlowVersion == null) throw new DSSErrorException(90011, "该工作流已经被删除，请重新创建"); - String userName = dssUserMapper.getuserName(dssFlowVersion.getUpdatorID()); - Map query = bmlService.query(userName, dssFlowVersion.getJsonPath(), dssFlowVersion.getVersion()); - dssFlowVersion.setJson(query.get("string").toString()); - dssFlow.setLatestVersion(dssFlowVersion); - return dssFlow; - } - - @Override - public DSSFlow getOneVersionFlow(Long flowID, String version, Long projectVersionID) { - DSSFlow dssFlow = getFlowByID(flowID); - DSSFlowVersion dssFlowVersion = flowMapper.selectVersionByFlowID(flowID, version, projectVersionID); - String userName = dssUserMapper.getuserName(dssFlowVersion.getUpdatorID()); - Map query = bmlService.query(userName, dssFlowVersion.getJsonPath(), dssFlowVersion.getVersion()); - dssFlowVersion.setJson(query.get("string").toString()); - dssFlow.setFlowVersions(Arrays.asList(dssFlowVersion)); - dssFlow.setLatestVersion(dssFlowVersion); - return dssFlow; - } - -/* @Override - public String getLatestJsonByFlow(DSSFlow DSSFlow) { - DSSFlow latestVersionFlow = getLatestVersionFlow(DSSFlow.getId()); - return latestVersionFlow.getLatestVersion().getJson(); - } - - @Override - public DSSFlow getLatestVersionFlow(Long ProjectID, String flowName) { - Long flowID = flowMapper.selectFlowIDByProjectIDAndFlowName(ProjectID, flowName); - return getLatestVersionFlow(flowID); - }*/ - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public void updateFlowBaseInfo(DSSFlow dssFlow, Long projectVersionID, Long taxonomyID) throws DSSErrorException { - try { - flowMapper.updateFlowBaseInfo(dssFlow); - } catch (DuplicateKeyException e) { - logger.info(e.getMessage()); - throw new DSSErrorException(90003, "工作流名不能重复"); - } - if (taxonomyID != null) updateFlowTaxonomyRelation(dssFlow.getId(), taxonomyID); - } - - @Override - public void updateFlowTaxonomyRelation(Long flowID, Long taxonomyID) throws DSSErrorException { - DSSFlow dssFlow = getFlowByID(flowID); - Long oldTaxonomyID = flowTaxonomyMapper.selectTaxonomyIDByFlowID(flowID); - if (!dssFlow.getRootFlow() && (!oldTaxonomyID.equals(taxonomyID))) throw new DSSErrorException(90010, "子工作流不允许更新分类id"); - if (!dssFlow.getRootFlow() && (oldTaxonomyID.equals(taxonomyID))) return; - //这里也要同时更新子工作流的分类id - List subFlowIDList = flowMapper.selectSubFlowIDByParentFlowID(flowID); - subFlowIDList.add(flowID); - flowTaxonomyMapper.updateFlowTaxonomyRelation(subFlowIDList, taxonomyID); - } - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public void batchDeleteFlow(List flowIDlist, Long projectVersionID) { - flowIDlist.stream().forEach(f -> { - deleteFlow(f, projectVersionID); - }); - } - - @Lock - @Transactional(rollbackFor = {DSSErrorException.class,AppJointErrorException.class}) - @Override - public String saveFlow(Long flowID, String jsonFlow, String comment, String userName, Long projectVersionID, List ops) throws DSSErrorException, AppJointErrorException { - for (Op op : ops) { - op.getParams().put("projectVersionID",projectVersionID); - op.getParams().put("userName",userName); - logger.info("{}保存工作流，操作:{}id:{}nodetype{},projectVersionID:{}",userName,op.getOp(),op.getId(),op.getNodeType(),projectVersionID); - Optional operate = Arrays.stream(operates).filter(f -> f.canInvokeOperate(op)).findFirst(); - // TODO: 2019/10/29 optinal 判断 and throws exception - switch (op.getOp()) { - case "add": - operate.get().add(this,op); - break; - case "update": - operate.get().update(this,op); - break; - case "delete": - operate.get().delete(this,op); - break; - default: - logger.error("other operation:unable to occur"); - } - } - //获取rsourceId，就是jsonPaht - String resourceId = getLatestVersionByFlowIDAndProjectVersionID(flowID, projectVersionID).getJsonPath(); - //上传文件获取resourceId和version save应该是已经有 - Map bmlReturnMap = bmlService.update(userName, resourceId, jsonFlow); - DSSFlowVersion dssFlowVersion = new DSSFlowVersion(); - dssFlowVersion.setUpdatorID(dssUserMapper.getUserID(userName)); - dssFlowVersion.setUpdateTime(new Date()); - dssFlowVersion.setVersion(bmlReturnMap.get("version").toString()); - dssFlowVersion.setJsonPath(resourceId); - dssFlowVersion.setComment(comment); - dssFlowVersion.setFlowID(flowID); - dssFlowVersion.setSource("保存更新"); - dssFlowVersion.setProjectVersionID(projectVersionID); - //version表中插入数据 - flowMapper.insertFlowVersion(dssFlowVersion); - return bmlReturnMap.get("version").toString(); - } - - @Override - public Integer getParentRank(Long parentFlowID) { - return getFlowByID(parentFlowID).getRank(); - } - - @Override - public DSSFlowVersion getLatestVersionByFlowIDAndProjectVersionID(Long flowID, Long projectVersionID) { - List versions = flowMapper.listVersionByFlowIDAndProjectVersionID(flowID, projectVersionID) - .stream().sorted((v1, v2) -> { - return v1.compareTo(v2); - }).collect(Collectors.toList()); - return versions.isEmpty() ? null : versions.get(0); - } - - @Override - public Long getParentFlowID(Long flowID) { - return flowMapper.getParentFlowID(flowID); - } - - @Deprecated - private Integer getParentFlowIDByFlowID(Long flowID, Integer rank) { - Long parentFlowID = flowMapper.selectParentFlowIDByFlowID(flowID); - if (parentFlowID != null) { - rank++; - getParentFlowIDByFlowID(parentFlowID, rank); - } - return rank; - } - - public void deleteFlow(Long flowId, Long projectVersionID) { - List subFlowIDs = flowMapper.selectSubFlowIDByParentFlowID(flowId); - for (Long subFlowID : subFlowIDs) { - deleteFlow(subFlowID, projectVersionID); - } - for (Long subFlowID : subFlowIDs) { - deleteDSSDB(subFlowID, projectVersionID); - // TODO: 2019/6/5 wtss发布过的工作流的删除？ - // TODO: 2019/6/5 json中资源的删除 - // TODO: 2019/6/5 事务的保证 - } - deleteDSSDB(flowId, projectVersionID); - } - - private void deleteDSSDB(Long flowID, Long projectVersionID) { - flowMapper.deleteFlowVersions(flowID, projectVersionID); - if (projectVersionID == null || (flowMapper.noVersions(flowID) != null && flowMapper.noVersions(flowID))) { - flowMapper.deleteFlowBaseInfo(flowID); - flowMapper.deleteFlowRelation(flowID); - flowTaxonomyMapper.deleteFlowTaxonomyRelation(flowID); - } - //第一期没有工作流的发布，所以不需要删除DSS工作流的发布表 - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSFlowTaxonomyServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSFlowTaxonomyServiceImpl.java deleted file mode 100644 index 0cbba745b..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSFlowTaxonomyServiceImpl.java +++ /dev/null @@ -1,138 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import com.webank.wedatasphere.dss.server.service.DSSFlowTaxonomyService; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.server.dao.FlowMapper; -import com.webank.wedatasphere.dss.server.dao.FlowTaxonomyMapper; -import com.webank.wedatasphere.dss.server.entity.DSSFlowTaxonomy; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlowVersion; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.server.lock.Lock; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.dao.DuplicateKeyException; -import org.springframework.stereotype.Service; -import org.springframework.transaction.annotation.Transactional; - -import java.util.Arrays; -import java.util.List; -import java.util.stream.Collectors; - - -@Service -public class DSSFlowTaxonomyServiceImpl implements DSSFlowTaxonomyService { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - @Autowired - private FlowTaxonomyMapper flowTaxonomyMapper; - - @Autowired - private FlowMapper flowMapper; - - @Autowired - private DSSProjectService projectService; - - @Autowired - private DSSFlowService flowService; - - @Override - public DSSFlowTaxonomy getFlowTaxonomyByID(Long id) { - return flowTaxonomyMapper.selectFlowTaxonomyByID(id); - } - - private List listFlowByTaxonomyID(Long projectID, Long taxonomyID, Boolean isRootFlow){ - return flowMapper.listFlowByTaxonomyID(projectID,taxonomyID,isRootFlow); - } - - // TODO: 2019/5/16 级联查询返回json序列化好像有问题，暂时只能循环查 - @Override - public List listAllFlowTaxonomy(Long projectVersionID, Boolean isRootFlow) { - DSSProject dssProject = projectService.getProjectByProjectVersionID(projectVersionID); - List dwsFlowTaxonomies = listFlowTaxonomyByProjectID(dssProject.getId()); - for (DSSFlowTaxonomy dssFlowTaxonomy : dwsFlowTaxonomies) { - List dssFlowList = listFlowByTaxonomyID(dssProject.getId(), dssFlowTaxonomy.getId(),isRootFlow); - for (DSSFlow dssFlow : dssFlowList) { - DSSFlowVersion version = flowService.getLatestVersionByFlowIDAndProjectVersionID(dssFlow.getId(),projectVersionID); - dssFlow.setLatestVersion(version); - } - dssFlowTaxonomy.setDssFlowList(dssFlowList.stream().filter(f ->f.getLatestVersion() !=null).collect(Collectors.toList())); - } - return dwsFlowTaxonomies; - } - - private List listFlowTaxonomyByProjectID(Long projectID) { - return flowTaxonomyMapper.listFlowTaxonomyByProjectID(projectID); - } - - //有projectID应该可以同时过滤不同用户中的，我的工作流分类下的工作流 - @Override - public List listFlowTaxonomy(Long projectVersionID, Long flowTaxonomyID, Boolean isRootFlow) { - DSSFlowTaxonomy flowTaxonomy = getFlowTaxonomyByID(flowTaxonomyID); - DSSProject dssProject = projectService.getProjectByProjectVersionID(projectVersionID); - List dssFlowList = listFlowByTaxonomyID(dssProject.getId(),flowTaxonomyID,isRootFlow); - for (DSSFlow dssFlow : dssFlowList) { - DSSFlowVersion version = flowService.getLatestVersionByFlowIDAndProjectVersionID(dssFlow.getId(),projectVersionID); - dssFlow.setLatestVersion(version); - } - flowTaxonomy.setDssFlowList(dssFlowList.stream().filter(f ->f.getLatestVersion() !=null).collect(Collectors.toList())); - return Arrays.asList(flowTaxonomy); - } - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public void addFlowTaxonomy(DSSFlowTaxonomy dssFlowTaxonomy, Long projectVersionID) throws DSSErrorException { - try { - flowTaxonomyMapper.insertFlowTaxonomy(dssFlowTaxonomy); - }catch (DuplicateKeyException e){ - logger.info(e.getMessage()); - throw new DSSErrorException(90005,"工作流名分类名不能重复"); - } - } - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public void updateFlowTaxonomy(DSSFlowTaxonomy dssFlowTaxonomy, Long projectVersionID) throws DSSErrorException { - try { - flowTaxonomyMapper.updateFlowTaxonomy(dssFlowTaxonomy); - }catch (DuplicateKeyException e){ - logger.info(e.getMessage()); - throw new DSSErrorException(90005,"工作流名分类名不能重复"); - } - } - - @Override - public boolean hasFlows(Long flowTaxonomyID) { - Long count = flowTaxonomyMapper.hasFlows(flowTaxonomyID); - return count != 0; - } - - @Lock - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public void deleteFlowTaxonomy(Long flowTaxonomyID,Long projectVersionID) { - flowTaxonomyMapper.deleteFlowTaxonomy(flowTaxonomyID); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSNodeInfoServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSNodeInfoServiceImpl.java deleted file mode 100644 index 0b9571f26..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSNodeInfoServiceImpl.java +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - -import com.webank.wedatasphere.dss.server.dao.NodeInfoMapper; -import com.webank.wedatasphere.dss.server.entity.NodeInfo; -import com.webank.wedatasphere.dss.server.service.DSSNodeInfoService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; - -import java.util.List; - - -@Service -public class DSSNodeInfoServiceImpl implements DSSNodeInfoService { - - @Autowired - private NodeInfoMapper nodeInfoMapper; - - @Override - public List getNodeType(String developProject) { - return nodeInfoMapper.selectNodeInfo(developProject); - } - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSProjectServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSProjectServiceImpl.java deleted file mode 100644 index 9d007cebc..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSProjectServiceImpl.java +++ /dev/null @@ -1,609 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.scheduler.SchedulerAppJoint; -import com.webank.wedatasphere.dss.appjoint.scheduler.entity.SchedulerProject; -import com.webank.wedatasphere.dss.appjoint.scheduler.hooks.ProjectPublishHook; -import com.webank.wedatasphere.dss.appjoint.scheduler.parser.ProjectParser; -import com.webank.wedatasphere.dss.appjoint.scheduler.tuning.ProjectTuning; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import com.webank.wedatasphere.dss.application.entity.Application; -import com.webank.wedatasphere.dss.application.service.ApplicationService; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlow; -import com.webank.wedatasphere.dss.common.entity.flow.DSSFlowVersion; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectPublishHistory; -import com.webank.wedatasphere.dss.common.entity.project.DSSProjectVersion; -import com.webank.wedatasphere.dss.common.entity.project.Project; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import com.webank.wedatasphere.dss.common.protocol.RequestDSSProject; -import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; -import com.webank.wedatasphere.dss.server.constant.DSSServerConstant; -import com.webank.wedatasphere.dss.server.dao.*; -import com.webank.wedatasphere.dss.server.entity.DSSFlowTaxonomy; -import com.webank.wedatasphere.dss.server.entity.DSSProjectTaxonomyRelation; -import com.webank.wedatasphere.dss.server.function.FunctionInvoker; -import com.webank.wedatasphere.dss.server.lock.Lock; -import com.webank.wedatasphere.dss.server.lock.LockEnum; -import com.webank.wedatasphere.dss.server.service.BMLService; -import com.webank.wedatasphere.dss.server.service.DSSFlowService; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.server.util.ThreadPoolTool; -import org.apache.commons.lang.StringUtils; -import org.apache.commons.math3.util.Pair; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; -import org.springframework.transaction.annotation.Transactional; - -import java.io.*; -import java.util.*; -import java.util.concurrent.ConcurrentHashMap; -import java.util.stream.Collectors; -import java.util.stream.Stream; - - -@Service -public class DSSProjectServiceImpl implements DSSProjectService { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private ProjectTaxonomyMapper projectTaxonomyMapper; - @Autowired - private DSSUserMapper dssUserMapper; - @Autowired - private FlowMapper flowMapper; - @Autowired - private FlowTaxonomyMapper flowTaxonomyMapper; - @Autowired - private DSSFlowService flowService; - @Autowired - private ProjectMapper projectMapper; - - @Autowired - private BMLService bmlService; - @Autowired - private ApplicationService applicationService; - @Autowired - private FunctionInvoker functionInvoker; - - private SchedulerAppJoint schedulerAppJoint=null; - - @Override - public DSSProject getProjectByID(Long id) { - /*JdbcProjectImpl instance = wtssdbConnector.getInjector().getInstance(JdbcProjectImpl.class); - Project project = instance.fetchProjectById(id.intValue()); - DSSProject DSSProject = EntityUtils.Project2DSSProject(project);*/ - return projectMapper.selectProjectByID(id); - } - - @Transactional(rollbackFor = {DSSErrorException.class,AppJointErrorException.class}) - @Override - public Long addProject(String userName, String name, String description, Long taxonomyID,String product,Integer applicationArea,String business, Long workspaceId) throws DSSErrorException, AppJointErrorException { - DSSProject dssProject = new DSSProject(); - dssProject.setUserName(userName); - dssProject.setName(name); - dssProject.setDescription(description); - //创建scheduler的project - if(existSchesulis()){ - createSchedulerProject(dssProject); - } - //创建appjoint的project - Map appjointProjectIDAndAppID = createAppjointProject(dssProject); - Long userID = dssUserMapper.getUserID(userName); - //创建dss自己的project - Pair pair = addDSSProject(userID, name, description,product,applicationArea,business, workspaceId); - //添加关联 - projectTaxonomyMapper.addProjectTaxonomyRelation(pair.getFirst(), taxonomyID, userID); - if(!appjointProjectIDAndAppID.isEmpty())projectMapper.addAccessProjectRelation(appjointProjectIDAndAppID,pair.getFirst()); - return pair.getSecond(); - } - - - - private Map createAppjointProject(DSSProject project) throws DSSErrorException, AppJointErrorException { - Map applicationProjectIDs = new HashMap(); - List> pairs = functionInvoker.projectServiceAddFunction(project, ProjectService::createProject, applicationService.listAppjoint()); - for (Pair pair : pairs) { - if(pair.getFirst().getId() != null){ - applicationProjectIDs.put(applicationService.getApplication(pair.getSecond()).getId(),pair.getFirst().getId()); - } - } - return applicationProjectIDs; - } - - private Pair addDSSProject(Long userID, String name, String description, String product, Integer applicationArea, String business, Long workspaceId) { - DSSProject dssProject = new DSSProject(); - dssProject.setUserID(userID); - dssProject.setName(name); - dssProject.setDescription(description); - dssProject.setSource(DSSServerConstant.DSS_PROJECT_SOURCE); - dssProject.setCreateTime(new Date()); - dssProject.setCreateBy(userID); - dssProject.setProduct(product); - dssProject.setApplicationArea(applicationArea); - dssProject.setBusiness(business); - dssProject.setWorkspaceId(workspaceId); - projectMapper.addProject(dssProject); - DSSProjectVersion dssProjectVersion = new DSSProjectVersion(); - dssProjectVersion.setComment(DSSServerConstant.DSS_PROJECT_FIRST_VERSION_COMMENT); - dssProjectVersion.setProjectID(dssProject.getId()); - dssProjectVersion.setUpdateTime(new Date()); - dssProjectVersion.setUpdatorID(userID); - dssProjectVersion.setVersion(DSSServerConstant.DSS_PROJECT_FIRST_VERSION); - dssProjectVersion.setLock(0); - projectMapper.addProjectVersion(dssProjectVersion); - return new Pair(dssProject.getId(), dssProjectVersion.getId()); - } - - private void createSchedulerProject(DSSProject dssProject) throws DSSErrorException { - try { - if(getSchedulerAppJoint() != null) { - functionInvoker.projectServiceAddFunction(dssProject, ProjectService::createProject, Arrays.asList(getSchedulerAppJoint())); - }else{ - logger.error("Add scheduler project failed for scheduler appjoint is null"); - } - } catch (Exception e) { - logger.error("add scheduler project failed,", e); - throw new DSSErrorException(90002, "add scheduler project failed" + e.getMessage()); - } - } - - @Override - public void updateProject(long projectID, String name, String description, String userName, String product ,Integer applicationArea ,String business) throws AppJointErrorException { - // TODO: 2019/9/19 appjoint的update - //无法修改名字 - //更新wtssProject中的description - if(!StringUtils.isBlank(description)){ - DSSProject project = getProjectByID(projectID); - project.setUserName(userName); - project.setDescription(description); - if(existSchesulis()){ - if(getSchedulerAppJoint() != null) { - functionInvoker.projectServiceFunction(project,ProjectService::updateProject,Arrays.asList(getSchedulerAppJoint())); - }else{ - logger.error("Update scheduler project failed for scheduler appjoint is null"); - } - } - functionInvoker.projectServiceFunction(project,ProjectService::updateProject,applicationService.listAppjoint()); - } - projectMapper.updateDescription(projectID, description, product ,applicationArea ,business); - } - - @Transactional(rollbackFor = DSSErrorException.class) - @Override - public void deleteProject(long projectID, String userName, Boolean ifDelScheduler) throws DSSErrorException { - try { - DSSProject project = getProjectByID(projectID); - project.setUserName(userName); - if(ifDelScheduler){ - if(existSchesulis()){ - if(getSchedulerAppJoint() != null) { - functionInvoker.projectServiceFunction(project, ProjectService::deleteProject, Arrays.asList(getSchedulerAppJoint())); - }else{ - logger.error("Delete scheduler project failed for scheduler appjoint is null"); - } - } - } - functionInvoker.projectServiceFunction(project,ProjectService::deleteProject,applicationService.listAppjoint()); - // TODO: 2019/11/15 删除relations表 - } catch (Exception e) { - logger.info("删除工程失败，原因：", e); - String errorMsg = e.getCause() == null ? e.getMessage() : e.getCause().getMessage(); - throw new DSSErrorException(90012, errorMsg); - } - flowTaxonomyMapper.deleteFlowTaxonomyByProjectID(projectID); - List dssFlowList = flowMapper.listFlowByProjectID(projectID); - flowService.batchDeleteFlow(dssFlowList.stream().map(f -> f.getId()).distinct().collect(Collectors.toList()), null); - projectMapper.deleteProjectVersions(projectID); - projectMapper.deleteProjectBaseInfo(projectID); - projectTaxonomyMapper.deleteProjectTaxonomyRelationByProjectID(projectID); - } - - @Override - public DSSProject getLatestVersionProject(Long projectID) { - DSSProject dssProject = getProjectByID(projectID); - DSSProjectVersion dssProjectVersion = projectMapper.selectLatestVersionByProjectID(projectID); - dssProject.setLatestVersion(dssProjectVersion); - return dssProject; - } - - /** - * 通过projectVersionID获取一个projrct对象，不会返回最新的版本信息 - * - * @param projectVersionID - * @return - */ - @Override - public DSSProject getProjectByProjectVersionID(Long projectVersionID) { - DSSProject dssProject = projectMapper.selectProjectByVersionID(projectVersionID); - return dssProject; - } - - @Override - public Boolean isPublished(Long projectID) { - return !projectMapper.noPublished(projectID); - } - - @Override - public List listAllProjectVersions(Long projectID) { - List dssProjectVersions = projectMapper.listProjectVersionsByProjectID(projectID); - for (DSSProjectVersion dssProjectVersion : dssProjectVersions) { - DSSProjectPublishHistory publishHistory = projectMapper.selectProjectPublishHistoryByProjectVersionID(dssProjectVersion.getId()); - dssProjectVersion.setPublishHistory(publishHistory); - } - return dssProjectVersions; - } - - - @Lock(type = LockEnum.ADD) - @Transactional(rollbackFor = {DSSErrorException.class, InterruptedException.class,AppJointErrorException.class}) - @Override - public void publish(Long projectVersionID, String userName, String comment) throws DSSErrorException, InterruptedException, AppJointErrorException { - - SchedulerAppJoint schedulerAppJoint = getSchedulerAppJoint(); - if(schedulerAppJoint != null) { - ProjectParser projectParser = schedulerAppJoint.getProjectParser(); - ProjectTuning projectTuning = schedulerAppJoint.getProjectTuning(); - ProjectPublishHook[] projectPublishHooks = schedulerAppJoint.getProjectPublishHooks(); - // TODO: 2019/9/24 try catch 下载json要挪到parser去 - //1.封装DSSProject - DSSProject dssProject = projectMapper.selectProjectByVersionID(projectVersionID); - dssProject.setUserName(dssUserMapper.getuserName(dssProject.getUserID())); - logger.info(userName + "-开始发布工程：" + dssProject.getName() + "版本ID为：" + projectVersionID); - ArrayList dssFlows = new ArrayList<>(); - List dssFlowVersionList = flowMapper.listLatestRootFlowVersionByProjectVersionID(projectVersionID); - for (DSSFlowVersion dssFlowVersion : dssFlowVersionList) { - DSSFlow dssFlow = flowMapper.selectFlowByID(dssFlowVersion.getFlowID()); - String json = (String) bmlService.query(userName, dssFlowVersion.getJsonPath(), dssFlowVersion.getVersion()).get("string"); - if (!dssFlow.getHasSaved()) { - logger.info("工作流{}从未保存过，忽略", dssFlow.getName()); - } else if (StringUtils.isNotBlank(json)) { - dssFlowVersion.setJson(json); - dssFlow.setLatestVersion(dssFlowVersion); - createPublishProject(userName, dssFlowVersion.getFlowID(), dssFlow, projectVersionID); - dssFlows.add(dssFlow); - } else { - String warnMsg = String.format(DSSServerConstant.PUBLISH_FLOW_REPORT_FORMATE, dssFlow.getName(), dssFlowVersion.getVersion()); - logger.info(warnMsg); - throw new DSSErrorException(90013, warnMsg); - } - } - if (dssFlows.isEmpty()) throw new DSSErrorException(90007, "该工程没有可以发布的工作流,请检查工作流是否都为空"); - dssProject.setFlows(dssFlows); - //2.封装DSSProject完成，开始发布 - SchedulerProject schedulerProject = projectParser.parseProject(dssProject); - projectTuning.tuningSchedulerProject(schedulerProject); - Stream.of(projectPublishHooks).forEach(DSSExceptionUtils.handling(hook -> hook.prePublish(schedulerProject))); - (schedulerAppJoint.getProjectService()).publishProject(schedulerProject, schedulerAppJoint.getSecurityService().login(userName)); - Stream.of(projectPublishHooks).forEach(DSSExceptionUtils.handling(hook -> hook.postPublish(schedulerProject))); - //3.发布完成后复制工程 - DSSProjectVersion dssProjectVersion = projectMapper.selectProjectVersionByID(projectVersionID); - copyProjectVersionMax(projectVersionID, dssProjectVersion, dssProjectVersion, userName, null); - }else { - logger.error("SchedulerAppJoint is null"); - throw new DSSErrorException(90014, "SchedulerAppJoint is null"); - } - } - - @Override - public Long createPublishHistory(String comment, Long creatorID, Long projectVersionID) { - DSSProjectPublishHistory dssProjectPublishHistory = new DSSProjectPublishHistory(); - dssProjectPublishHistory.setComment(comment); - dssProjectPublishHistory.setCreateID(creatorID); - dssProjectPublishHistory.setCreateTime(new Date()); - dssProjectPublishHistory.setUpdateTime(new Date()); - dssProjectPublishHistory.setProjectVersionID(projectVersionID); - dssProjectPublishHistory.setState(0); - projectMapper.insertPublishHistory(dssProjectPublishHistory); - return dssProjectPublishHistory.getId(); - } - - @Override - public void updatePublishHistory(Long projectVersionID, Integer status, Date updateTime) { - projectMapper.updatePublishHistoryState(projectVersionID, status); - } - - @Override - public DSSProjectPublishHistory getPublishHistoryByID(Long projectVersionID) { - return projectMapper.selectProjectPublishHistoryByProjectVersionID(projectVersionID); - } - - @Override - public DSSProject getExecutionDSSProject(RequestDSSProject requestDSSProject) throws DSSErrorException { - DSSFlow dssFlow = flowService.getOneVersionFlow(requestDSSProject.flowId(), requestDSSProject.version(), requestDSSProject.projectVersionId()); - DSSProject dssProject = projectMapper.selectProjectByVersionID(requestDSSProject.projectVersionId()); - dssProject.setUserName(dssUserMapper.getuserName(dssProject.getUserID())); - DSSFlow returnFlow = recursiveGenerateParentFlow(dssFlow, requestDSSProject); - dssProject.setFlows(Arrays.asList(returnFlow)); - return dssProject; - } - - @Override - public Long getAppjointProjectID(Long projectID, String nodeType) { - // TODO: 2019/11/15 判断工程在dss中存在 - Application applicationbyNodeType = applicationService.getApplicationbyNodeType(nodeType); - Long appjointProjectID = projectMapper.getAppjointProjectID(projectID, applicationbyNodeType.getId()); - return appjointProjectID == null? projectID:appjointProjectID; - } - - @Override - public Long getAppjointProjectIDByApplicationName(Long projectID, String applicationName) { - Long appjointProjectID = projectMapper.getAppjointProjectID(projectID, applicationService.getApplication(applicationName).getId()); - return appjointProjectID == null? projectID:appjointProjectID; - } - - private DSSFlow recursiveGenerateParentFlow(DSSFlow dssFlow, RequestDSSProject requestDSSProject) throws DSSErrorException { - DSSFlow returnFlow = null; - Long parentFlowID = flowService.getParentFlowID(dssFlow.getId()); - if(parentFlowID != null){ - //对于当前执行的工作流的父工作流，直接找其最新的版本 - DSSFlow parentFlow = flowService.getLatestVersionFlow(parentFlowID, requestDSSProject.projectVersionId()); - parentFlow.setChildren(Arrays.asList(dssFlow)); - returnFlow = recursiveGenerateParentFlow(parentFlow, requestDSSProject); - }else { - returnFlow = dssFlow; - } - return returnFlow; - } - - private void createPublishProject(String userName, Long parentFlowID, DSSFlow dssFlowParent, Long projectVersionID) throws DSSErrorException { - List subFlowIDS = flowMapper.selectSubFlowIDByParentFlowID(parentFlowID); - ArrayList dssFlows = new ArrayList<>(); - for (Long subFlowID : subFlowIDS) { - DSSFlowVersion dssFlowVersion = flowService.getLatestVersionByFlowIDAndProjectVersionID(subFlowID, projectVersionID); - if (dssFlowVersion != null) { //subFlowIDS通过flow关联查出来的，但是有可能最新版本的project对已有的flows做了删除 - DSSFlow dssFlow = flowMapper.selectFlowByID(dssFlowVersion.getFlowID()); - String json = (String) bmlService.query(userName, dssFlowVersion.getJsonPath(), dssFlowVersion.getVersion()).get("string"); - if (!dssFlow.getHasSaved()) { - logger.info("工作流{}从未保存过，忽略", dssFlow.getName()); - }else if (StringUtils.isNotBlank(json)){ - dssFlowVersion.setJson(json); - dssFlow.setLatestVersion(dssFlowVersion); - createPublishProject(userName, subFlowID, dssFlow, projectVersionID); - dssFlows.add(dssFlow); - } else { - String warnMsg = String.format(DSSServerConstant.PUBLISH_FLOW_REPORT_FORMATE, dssFlow.getName(), dssFlowVersion.getVersion()); - logger.info(warnMsg); - throw new DSSErrorException(90013, warnMsg); - } - } - } - dssFlowParent.setChildren(dssFlows); - } - - /* - 复制工程 - */ - @Lock - @Transactional(rollbackFor = {DSSErrorException.class, InterruptedException.class,AppJointErrorException.class}) - @Override - public Long copyProject(Long projectVersionID, Long projectID, String projectName, String userName) throws DSSErrorException, InterruptedException, AppJointErrorException { - DSSProject project = projectMapper.selectProjectByID(projectID); - if (StringUtils.isNotEmpty(projectName)) {project.setName(projectName);} - DSSProjectTaxonomyRelation projectTaxonomyRelation = projectTaxonomyMapper.selectProjectTaxonomyRelationByTaxonomyIdOrProjectId(projectID); - //添加至wtss的project数据库，获取projectID - project.setUserName(userName); - if(existSchesulis()){ - createSchedulerProject(project); - } - Map appjointProjectIDAndAppID = createAppjointProject(project); - Long userID = dssUserMapper.getUserID(userName); - //添加至DSS的project数据库，这里的projectID应该不需要自增 - //目前是相同数据库，需要自增id - project.setUserID(userID); - project.setCreateTime(new Date()); - project.setId(null); - projectMapper.addProject(project); - if(!appjointProjectIDAndAppID.isEmpty())projectMapper.addAccessProjectRelation(appjointProjectIDAndAppID,project.getId()); - projectTaxonomyMapper.addProjectTaxonomyRelation(project.getId(), projectTaxonomyRelation.getTaxonomyId(), userID); - DSSProjectVersion maxVersion = projectMapper.selectLatestVersionByProjectID(projectID); - copyProjectVersionMax(maxVersion.getId(), maxVersion, maxVersion, userName, project.getId()); - return project.getId(); - } - - private boolean existSchesulis(){ - return applicationService.getApplication("schedulis") != null; - } - - - /** - * 复制工程的某个历史版本到最新版本,同时复制工作流 - * - * @param projectVersionID - * @param copyVersion 复制的工程版本 - * @param WTSSprojectID 如果是工程复制,则使用此参数. 指定复制工作流的初始版本号. - * 如果是工程版本复制,则设置为null即可. - */ - @Lock(type = LockEnum.ADD) - @Transactional(rollbackFor = {DSSErrorException.class, InterruptedException.class}) - @Override - public void copyProjectVersionMax(Long projectVersionID, DSSProjectVersion maxVersion, DSSProjectVersion copyVersion, String userName, Long WTSSprojectID) throws DSSErrorException, InterruptedException { -// copy project_version - String maxVersionNum = generateNewVersion(maxVersion.getVersion()); - if (null != WTSSprojectID) { - copyVersion.setVersion(DSSServerConstant.DSS_PROJECT_FIRST_VERSION); - copyVersion.setProjectID(WTSSprojectID); - } else { - copyVersion.setVersion(maxVersionNum); - } - Long userID = dssUserMapper.getUserID(userName); - copyVersion.setUpdatorID(userID); - copyVersion.setUpdateTime(new Date()); - List flowVersions = flowMapper.listLastFlowVersionsByProjectVersionID(copyVersion.getId()) - .stream().sorted((o1, o2) -> Integer.valueOf(o1.getFlowID().toString()) - Integer.valueOf(o2.getFlowID().toString())) - .collect(Collectors.toList()); - Long oldProjectVersionID = copyVersion.getId(); - copyVersion.setId(null); - projectMapper.addProjectVersion(copyVersion); - if (copyVersion.getId() == null) {throw new DSSErrorException(90015, "复制工程版本失败");} - Map subAndParentFlowIDMap = new ConcurrentHashMap<>(); - // copy flow - if (null != WTSSprojectID) { - flowVersions.stream().forEach(f -> { - DSSFlow flow = flowMapper.selectFlowByID(f.getFlowID()); - Long parentFlowID = flowMapper.selectParentFlowIDByFlowID(flow.getId()); - if (parentFlowID != null) {subAndParentFlowIDMap.put(flow.getId(), parentFlowID);} - }); - for (DSSFlowVersion fv : flowVersions) { - // 添加所有父子到map中 - DSSFlow flow = flowMapper.selectFlowByID(fv.getFlowID()); - flow.setCreatorID(userID); - flow.setName(flow.getName()); - flow.setProjectID(copyVersion.getProjectID()); - flow.setCreateTime(new Date()); - - Long taxonomyID = flowTaxonomyMapper.selectTaxonomyIDByFlowID(flow.getId()); - DSSFlowTaxonomy flowTaxonomy = flowTaxonomyMapper.selectFlowTaxonomyByID(taxonomyID); - //新增flow相关数据 - fv.setOldFlowID(flow.getId()); - flow.setId(null); - flowMapper.insertFlow(flow); - if (null == flow.getId()) {throw new DSSErrorException(90016, "复制工作流失败");} - for (Map.Entry entry : subAndParentFlowIDMap.entrySet()) { - if (entry.getValue().equals(fv.getFlowID())){subAndParentFlowIDMap.put(entry.getKey(), flow.getId());} - if (entry.getKey().equals(fv.getFlowID())){subAndParentFlowIDMap.put(flow.getId(), entry.getValue());} - } - if (flowTaxonomy.getProjectID() != -1 && (!flowTaxonomy.getProjectID().equals(copyVersion.getProjectID()))) { - flowTaxonomy.setProjectID(copyVersion.getProjectID()); - flowTaxonomy.setCreateTime(new Date()); - flowTaxonomy.setUpdateTime(new Date()); - flowTaxonomy.setProjectID(copyVersion.getUpdatorID()); - flowTaxonomyMapper.insertFlowTaxonomy(flowTaxonomy); - } - if (null != taxonomyID){flowTaxonomyMapper.insertFlowTaxonomyRelation(flowTaxonomy.getId(), flow.getId());} - fv.setFlowID(flow.getId()); - } - for (DSSFlowVersion fv : flowVersions) { - if (subAndParentFlowIDMap.get(fv.getFlowID()) != null){flowMapper.insertFlowRelation(fv.getFlowID(), subAndParentFlowIDMap.get(fv.getFlowID()));} - } - } -// copy flow_version - if (flowVersions.size() > 0) { - ThreadPoolTool tool = new ThreadPoolTool(5, flowVersions); - tool.setCallBack(new ThreadPoolTool.CallBack() { - @Override - public void method(List flowVersions) { - for (DSSFlowVersion fv : flowVersions) { - // 工作流版本的json文件，都是需要重新上传到bml - Map bmlQueryMap = bmlService.download(dssUserMapper.getuserName(fv.getUpdatorID()), fv.getJsonPath(), fv.getVersion()); - BufferedReader bufferedReader = new BufferedReader(new InputStreamReader((InputStream) bmlQueryMap.get("is"))); - StringBuilder sb = new StringBuilder(); - String s = null; - try { - while ((s = bufferedReader.readLine()) != null) { - sb.append(s); - sb.append("\n"); - } - String json = sb.toString().trim(); - List newflowIDs = getSubNewFlowID(subAndParentFlowIDMap, fv.getFlowID()); - List oldFlowIDs = getOldSubFlowID(subAndParentFlowIDMap, fv.getFlowID()); - if (oldFlowIDs != null && newflowIDs != null) { - logger.info("replace之前:" + json); - if (json != null) { - for (int i = 0; i < newflowIDs.size(); i++) { - json = json.replace(DSSServerConstant.EMVEDDEDFLOWID + oldFlowIDs.get(i), DSSServerConstant.EMVEDDEDFLOWID + newflowIDs.get(i)); - } - } - logger.info("replace之后:" + json); - } - if (json != null) { - InputStream in_nocode = new ByteArrayInputStream(json.getBytes()); - Map bmlReturnMap = bmlService.upload(userName, in_nocode, UUID.randomUUID().toString() + ".json"); - fv.setProjectVersionID(copyVersion.getId()); - fv.setUpdateTime(new Date()); - fv.setUpdatorID(copyVersion.getUpdatorID()); - String oldFlowVersion = fv.getVersion(); - fv.setJsonPath(bmlReturnMap.get("resourceId").toString()); - fv.setVersion(bmlReturnMap.get("version").toString()); - fv.setSource("Copy from Old { ProjectVersionID : " + oldProjectVersionID + " ,FlowID:" + fv.getOldFlowID() + ",Version:" + oldFlowVersion + "} to New { ProjectVersionID:" + fv.getProjectVersionID() + " ,FlowID:" + fv.getFlowID() + ",Version:" + fv.getVersion() + "}"); - bufferedReader.close(); - in_nocode.close(); - } - } catch (IOException e) { - logger.error("工作流复制IO异常", e); - } - - } - } - }); - tool.excute(); - } - if (flowVersions != null && flowVersions.size() > 0) { - flowVersions.stream().forEach(f -> { - logger.info("jsonPaht:{},oldeFlowID:{},newFlowID", f.getJsonPath(), f.getOldFlowID(), f.getFlowID()); - }); - flowMapper.batchInsertFlowVersion(flowVersions); - } - } - - private String generateNewVersion(String version){ - int next = Integer.parseInt(version.substring(1, version.length())) + 1; - return DSSServerConstant.VERSION_PREFIX + String.format(DSSServerConstant.VERSION_FORMAT, next); - } - - - private List getOldSubFlowID(Map subAndParentFlowIDMap, Long flowID) { - List> collect = subAndParentFlowIDMap.entrySet().stream().filter(f -> f.getValue().equals(flowID)).collect(Collectors.toList()); - if (collect.isEmpty()) return null; - int size = collect.size() / 2; - ArrayList longs = new ArrayList<>(); - List> sortedCollections = collect.stream().sorted((o, n) -> { - return Integer.valueOf(o.getKey().toString()) - Integer.valueOf(n.getKey().toString()); - }).collect(Collectors.toList()); - sortedCollections.subList(0, size).stream().forEach(p -> longs.add(p.getKey())); - return longs; - /*if(collect.isEmpty()){ - return UUID.randomUUID().toString(); - }else { - return collect.get(0).getKey() - collect.get(1).getKey() <0? Constant.EMVEDDEDFLOWID +collect.get(0).getKey():Constant.EMVEDDEDFLOWID +collect.get(1).getKey(); - }*/ - } - - private List getSubNewFlowID(Map subAndParentFlowIDMap, Long flowID) { - List> collect = subAndParentFlowIDMap.entrySet().stream().filter(f -> f.getValue().equals(flowID)).collect(Collectors.toList()); - if (collect.isEmpty()) return null; - int size = collect.size() / 2; - ArrayList longs = new ArrayList<>(); - List> sortedCollections = collect.stream().sorted((o, n) -> { - return Integer.valueOf(n.getKey().toString()) - Integer.valueOf(o.getKey().toString()); - }).collect(Collectors.toList()); - sortedCollections.subList(0, size).stream().forEach(p -> longs.add(p.getKey())); - return longs.stream().sorted((o1, o2) -> { - return Integer.valueOf(o1.toString()) - Integer.valueOf(o2.toString()); - }).collect(Collectors.toList()); - } - - - private SchedulerAppJoint getSchedulerAppJoint(){ - if(schedulerAppJoint == null){ - try { - schedulerAppJoint = (SchedulerAppJoint)applicationService.getAppjoint("schedulis"); - } catch (AppJointErrorException e) { - logger.error("Schedule system init failed!", e); - } - } - return schedulerAppJoint; - } - -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSProjectTaxonomyServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSProjectTaxonomyServiceImpl.java deleted file mode 100644 index 215115167..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSProjectTaxonomyServiceImpl.java +++ /dev/null @@ -1,132 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - - -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.server.dao.ProjectTaxonomyMapper; -import com.webank.wedatasphere.dss.server.entity.DSSProjectTaxonomy; -import com.webank.wedatasphere.dss.server.service.DSSProjectService; -import com.webank.wedatasphere.dss.server.service.DSSProjectTaxonomyService; -import com.webank.wedatasphere.dss.common.exception.DSSErrorException; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.dao.DuplicateKeyException; -import org.springframework.stereotype.Service; - -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; - - -@Service -public class DSSProjectTaxonomyServiceImpl implements DSSProjectTaxonomyService { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - @Autowired - private ProjectTaxonomyMapper projectTaxonomyMapper; - @Autowired - private DSSProjectService projectService; - - @Override - public DSSProjectTaxonomy getProjectTaxonomyByID(Long id) { - return projectTaxonomyMapper.selectProjectTaxonomyByID(id); - } - - @Override - public List listProjectTaxonomyByUser(String userName) { - return projectTaxonomyMapper.listProjectTaxonomyByUser(userName); - } - - - private List listProjectIDByTaxonomyID(Long taxonomyID, String userName) { - return projectTaxonomyMapper.listProjectIDByTaxonomyID(taxonomyID, userName); - } - - - @Override - public List listAllProjectTaxonomy(String userName, Long workspaceId) { - List dssProjectTaxonomies = listProjectTaxonomyByUser(userName); - for (DSSProjectTaxonomy dssProjectTaxonomy : dssProjectTaxonomies) { - List projectIDs = listProjectIDByTaxonomyID(dssProjectTaxonomy.getId(), userName); - ArrayList dssProjectList = new ArrayList<>(); - for (Long projectID : projectIDs) { - DSSProject dssProject = projectService.getLatestVersionProject(projectID); - // 只选择返回属于这个workspace的project，(某些用户拥有多个workspace的不同project) - if(workspaceId.equals(dssProject.getWorkspaceId())) { - dssProjectList.add(dssProject); - } - } - dssProjectTaxonomy.setDssProjectList(dssProjectList); - } - return dssProjectTaxonomies; - } - - @Override - public List listProjectTaxonomy(Long taxonomyID, String userName) { - DSSProjectTaxonomy dssProjectTaxonomy = getProjectTaxonomyByID(taxonomyID); - List projectIDs = listProjectIDByTaxonomyID(dssProjectTaxonomy.getId(), userName); - ArrayList dssProjectList = new ArrayList<>(); - for (Long projectID : projectIDs) { - DSSProject dssProject = projectService.getLatestVersionProject(projectID); - dssProjectList.add(dssProject); - } - dssProjectTaxonomy.setDssProjectList(dssProjectList); - return Arrays.asList(dssProjectTaxonomy); - } - - @Override - public void addProjectTaxonomy(DSSProjectTaxonomy dssProjectTaxonomy) throws DSSErrorException { - try { - projectTaxonomyMapper.insertProjectTaxonomy(dssProjectTaxonomy); - } catch (DuplicateKeyException e) { - logger.info(e.getMessage()); - throw new DSSErrorException(90004, "工程分类名不能重复"); - } - } - - @Override - public void updateProjectTaxonomy(DSSProjectTaxonomy dssProjectTaxonomy) throws DSSErrorException { - try { - projectTaxonomyMapper.updateProjectTaxonomy(dssProjectTaxonomy); - } catch (DuplicateKeyException e) { - logger.info(e.getMessage()); - throw new DSSErrorException(90004, "工程分类名不能重复"); - } - } - - @Override - public boolean hasProjects(Long projectTaxonomyID) { - Long count = projectTaxonomyMapper.hasProjects(projectTaxonomyID); - return count != 0; - } - - @Override - public void deleteProjectTaxonomy(Long projectTaxonomyID) { - projectTaxonomyMapper.deleteProjectTaxonomy(projectTaxonomyID); - } - - @Override - public void updateProjectTaxonomyRelation(Long projectID, Long taxonomyID) { - if (taxonomyID != null) { - projectTaxonomyMapper.updateProjectTaxonomyRelation(projectID, taxonomyID); - } - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSUserServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSUserServiceImpl.java deleted file mode 100644 index 60973c98f..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSUserServiceImpl.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - - -import com.webank.wedatasphere.dss.application.entity.DSSUser; -import com.webank.wedatasphere.dss.server.dao.DSSUserMapper; -import com.webank.wedatasphere.dss.server.service.DSSUserService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; - - -@Service -public class DSSUserServiceImpl implements DSSUserService { - @Autowired - private DSSUserMapper dssUserMapper; - - @Override - public Long getUserID(String userName) { - return dssUserMapper.getUserID(userName); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSWorkspaceServiceImpl.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSWorkspaceServiceImpl.java deleted file mode 100644 index 19a27befb..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/service/impl/DSSWorkspaceServiceImpl.java +++ /dev/null @@ -1,155 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service.impl; - -import com.webank.wedatasphere.dss.server.constant.DSSServerConstant; -import com.webank.wedatasphere.dss.server.dao.WorkspaceMapper; -import com.webank.wedatasphere.dss.server.dto.response.*; -import com.webank.wedatasphere.dss.server.entity.*; -import com.webank.wedatasphere.dss.server.service.DSSWorkspaceService; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; - -import java.util.ArrayList; -import java.util.List; - -/** - * Created by schumiyi on 2020/6/22 - */ -@Service -public class DSSWorkspaceServiceImpl implements DSSWorkspaceService { - - @Autowired - private WorkspaceMapper workspaceMapper; - - @Override - public List getWorkspaces() { - - return workspaceMapper.getWorkspaces(); - } - - @Override - public DSSWorkspace getWorkspacesById(Long id) { - return workspaceMapper.getWorkspaceById(id); - } - - - @Override - public Long addWorkspace(String userName, String name, String department, String label, String description) { - DSSWorkspace dssWorkspace = new DSSWorkspace(); - dssWorkspace.setName(name); - dssWorkspace.setDepartment(department); - dssWorkspace.setDescription(description); - dssWorkspace.setLabel(label); - dssWorkspace.setCreateBy(userName); - dssWorkspace.setSource(DSSServerConstant.DSS_WORKSPACE_SOURCE); - dssWorkspace.setLastUpdateUser(userName); - workspaceMapper.addWorkSpace(dssWorkspace); - return dssWorkspace.getId(); - } - - @Override - public boolean existWorkspaceName(String name) { - return !workspaceMapper.findByWorkspaceName(name).isEmpty(); - } - - @Override - public List getWorkSpaceDepartments() { - // TODO: service层和dao层完善 - WorkspaceDepartmentVo dp = new WorkspaceDepartmentVo(); - dp.setId(1L); - dp.setName("应用开发组"); - WorkspaceDepartmentVo di = new WorkspaceDepartmentVo(); - di.setId(2L); - di.setName("平台研发组"); - List departments = new ArrayList<>(); - departments.add(dp); - departments.add(di); - return departments; - } - - @Override - public List getHomepageDemos(boolean isChinese) { - List demoMenuVos = isChinese ? workspaceMapper.getHomepageDemoMenusCn() : workspaceMapper.getHomepageDemoMenusEn(); - for (HomepageDemoMenuVo demoMenuVo : demoMenuVos) { - Long menuId = demoMenuVo.getId(); - List demoInstanceVos = isChinese ? workspaceMapper.getHomepageInstancesByMenuIdCn(menuId) : workspaceMapper.getHomepageInstancesByMenuIdEn(menuId); - demoMenuVo.setDemoInstances(demoInstanceVos); - } - return demoMenuVos; - } - - @Override - public List getHomepageVideos(boolean isChinese) { - return isChinese ? workspaceMapper.getHomepageVideosCn() : workspaceMapper.getHomepageVideosEn(); - } - - @Override - public List getWorkspaceManagements(Long workspaceId, String username, boolean isChinese) { - if (!isAdminUser(workspaceId, username)) { - return new ArrayList<>(); - } - List managementMenuVos = isChinese ? workspaceMapper.getManagementMenuCn() : workspaceMapper.getManagementMenuEn(); - return getMenuAppInstances(managementMenuVos, isChinese); - } - - private List getMenuAppInstances(List menuVos, boolean isChinese) { - for (OnestopMenuVo menuVo : menuVos) { - Long menuId = menuVo.getId(); - List menuAppInstanceVos = isChinese ? workspaceMapper.getMenuAppInstancesCn(menuId) : workspaceMapper.getMenuAppInstancesEn(menuId); - menuVo.setAppInstances(menuAppInstanceVos); - } - return menuVos; - } - - @Override - public List getWorkspaceApplications(Long workspaceId, String username, boolean isChinese) { - List applicationMenuVos = isChinese ? workspaceMapper.getApplicationMenuCn() : workspaceMapper.getApplicationMenuEn(); - return getMenuAppInstances(applicationMenuVos, isChinese); - } - - @Override - public List getWorkspaceFavorites(Long workspaceId, String username, boolean isChinese) { - return isChinese ? workspaceMapper.getWorkspaceFavoritesCn(username, workspaceId) : workspaceMapper.getWorkspaceFavoritesEn(username, workspaceId); - } - - @Override - public Long addFavorite(String username, Long workspaceId, Long menuApplicationId) { - DSSFavorite dssFavorite = new DSSFavorite(); - dssFavorite.setUsername(username); - dssFavorite.setWorkspaceId(workspaceId); - dssFavorite.setMenuApplicationId(menuApplicationId); - // todo: order will from the front end - dssFavorite.setOrder(1); - dssFavorite.setCreateBy(username); - dssFavorite.setLastUpdateUser(username); - workspaceMapper.addFavorite(dssFavorite); - return dssFavorite.getId(); - } - - @Override - public Long deleteFavorite(String username, Long favouritesId) { - workspaceMapper.deleteFavorite(favouritesId); - return favouritesId; - } - - private boolean isAdminUser(Long workspaceId, String username) { - DSSWorkspace workspace = workspaceMapper.getWorkspaceById(workspaceId); - return username != null && workspace != null && username.equals(workspace.getCreateBy()); - } -} diff --git a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/util/ThreadPoolTool.java b/dss-server/src/main/java/com/webank/wedatasphere/dss/server/util/ThreadPoolTool.java deleted file mode 100644 index fe0f40fc8..000000000 --- a/dss-server/src/main/java/com/webank/wedatasphere/dss/server/util/ThreadPoolTool.java +++ /dev/null @@ -1,132 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.util; - - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - - -import java.util.List; -import java.util.concurrent.CountDownLatch; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; - -public class ThreadPoolTool { - - private Logger logger = LoggerFactory.getLogger(this.getClass()); - - //单个线程处理的数据量 - private int singleCount; - //处理的总数据量 - private int listSize; - //开启的线程数 - private int runSize; - //操作的数据集 - private List list; - //计数器 - private CountDownLatch begin,end; - //线程池 - private ExecutorService executorService; - //回调 - private CallBack callBack; - - - public void setCallBack(CallBack callBack) { - this.callBack = callBack; - } - - public ThreadPoolTool(int singleCount, List list){ - this.singleCount = singleCount; - this.list = list; - if (list != null){ - this.listSize = list.size(); - this.runSize = (this.listSize/this.singleCount) + 1; - } - } - - public void excute() throws InterruptedException { - executorService = Executors.newFixedThreadPool(runSize); - begin = new CountDownLatch(1); - end = new CountDownLatch(runSize); - //创建线程 - int startIndex = 0; - int endIndex = 0; - List newList = null; - for (int i = 0; i < runSize; i++) { - //计算每个线程对应的数据 - if (i < (runSize - 1)){ - startIndex = i * singleCount; - endIndex = (i + 1) * singleCount; - newList = list.subList(startIndex,endIndex); - }else { - startIndex = i * singleCount; - endIndex = listSize; - newList = list.subList(startIndex,endIndex); - } - //创建线程类处理数据 - MyThread myThread = new MyThread(newList, begin, end) { - @Override - public void method(List list) { - callBack.method(list); - } - }; - //执行线程 - executorService.execute(myThread); - } - //计数器减一 - begin.countDown(); - end.await(); - //关闭线程池 - executorService.shutdown(); - } - - //抽象线程类 - public abstract class MyThread implements Runnable{ - - private List list; - private CountDownLatch begin,end; - - public MyThread(List list, CountDownLatch begin, CountDownLatch end){ - this.list = list; - this.begin = begin; - this.end = end; - } - - @Override - public void run() { - try { - //执行程序 - method(list); - begin.await(); - } catch (InterruptedException e) { - logger.error(e.getMessage()); - Thread.currentThread().interrupt(); - }finally { - //计数器减一 - end.countDown(); - } - } - public abstract void method(List list); - } - - //回调接口定义 - public interface CallBack{ - public void method(List list); - } -} diff --git a/dss-server/src/main/resources/application.yml b/dss-server/src/main/resources/application.yml deleted file mode 100644 index 419506c3f..000000000 --- a/dss-server/src/main/resources/application.yml +++ /dev/null @@ -1,22 +0,0 @@ -server: - port: 9004 -spring: - application: - name: dss-server - - -eureka: - client: - serviceUrl: - defaultZone: http://127.0.0.1:20303/eureka/ - instance: - metadata-map: - test: wedatasphere - -management: - endpoints: - web: - exposure: - include: refresh,info -logging: - config: classpath:log4j2.xml diff --git a/dss-server/src/main/resources/default/AddschedulerUser.sh b/dss-server/src/main/resources/default/AddschedulerUser.sh deleted file mode 100644 index 116c2d96b..000000000 --- a/dss-server/src/main/resources/default/AddschedulerUser.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -user=$1 -password=$2 -installDir=$3 - - -if grep -i "^${user}=" $installDir/token.properties; - then - sed -i '' "s/^$user=.*/$user=$password/" $installDir/token.properties -else - echo "$user=$password" >> $installDir/token.properties -fi diff --git a/dss-server/src/main/resources/default/CreateLdapAccount.sh b/dss-server/src/main/resources/default/CreateLdapAccount.sh deleted file mode 100644 index 1318ab16b..000000000 --- a/dss-server/src/main/resources/default/CreateLdapAccount.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -source /etc/profile -server_host=$1 -server_login_user=$2 -server_login_password=$3 -server_python_path=$4 -server_ldap_source_path=$5 -ldap_user=$6 -ldap_password=$7 -echo "$server_login_password ssh $server_login_user@$server_host sudo python $server_python_path add_with_pw $ldap_user -p $ldap_password" -sshpass -p $server_login_password ssh $server_login_user@$server_host "sudo python $server_python_path add_with_pw $ldap_user -p $ldap_password" -#sshpass -p $server_login_password ssh $server_login_user@$server_host "sudo su - root -c 'source /etc/profile && source $server_ldap_source_path && sudo python $server_python_path add_with_pw $ldap_user -p $ldap_password && deactivate'" - -echo "******************LDAP USER CREATED***********************" - - - - diff --git a/dss-server/src/main/resources/default/CreateLinuxUser.sh b/dss-server/src/main/resources/default/CreateLinuxUser.sh deleted file mode 100644 index fabcac1c5..000000000 --- a/dss-server/src/main/resources/default/CreateLinuxUser.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash - -source /etc/profile -server_user_strs=$1 -echo $server_user_strs -add_user_name=$2 -add_user_password=$3 -server_user_array=(${server_user_strs//,/ }) -for server_user_str in ${server_user_array[@]} -do - server_user_info=(${server_user_str//#/ }) - server_host=${server_user_info[0]} - server_user_name=${server_user_info[1]} - server_user_password=${server_user_info[2]} - echo "${server_host},${server_user_name},${server_user_password}" - - sudo sshpass -p $server_user_password ssh -o ConnectTimeout=1 $server_user_name@$server_host "echo success" - [ $? -ne 0 ] && echo "登录主机${server_host}失败" && exit 254 -done - -echo "************服务器网络校验通过,开始创建用户*****************" - -for server_user_str in ${server_user_array[@]} -do - - server_user_info=(${server_user_str//#/ }) - server_host=${server_user_info[0]} - server_user_name=${server_user_info[1]} - server_user_password=${server_user_info[2]} - - #sshpass -p $server_user_password ssh $server_user_name@$server_host "sudo useradd $add_user_name && echo $add_user_password |sudo -i passwd --stdin $add_user_name" - sshpass -p $server_user_password ssh $server_user_name@$server_host "sudo useradd $add_user_name -s /sbin/nologin" - - [ $? -ne 0 ] && echo "创建用户失败：${host}失败" && exit 254 -done diff --git a/dss-server/src/main/resources/default/HdfsPath.sh b/dss-server/src/main/resources/default/HdfsPath.sh deleted file mode 100644 index cd765486c..000000000 --- a/dss-server/src/main/resources/default/HdfsPath.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash - -user=$1 -dir=$2 -hdfs dfs -mkdir -p $dir -hdfs dfs -chown $user:$user $dir \ No newline at end of file diff --git a/dss-server/src/main/resources/default/LinuxPath.sh b/dss-server/src/main/resources/default/LinuxPath.sh deleted file mode 100644 index bc55860b1..000000000 --- a/dss-server/src/main/resources/default/LinuxPath.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -user=$1 -dir=$2 -echo $1 $2; -sudo mkdir -p $dir -sudo chown $user:$user $dir \ No newline at end of file diff --git a/dss-server/src/main/resources/linkis.properties b/dss-server/src/main/resources/linkis.properties deleted file mode 100644 index 97be7db01..000000000 --- a/dss-server/src/main/resources/linkis.properties +++ /dev/null @@ -1,71 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -wds.linkis.test.mode=true - -wds.linkis.server.mybatis.datasource.url=jdbc:mysql://0.0.0.1:3306/linkis?characterEncoding=UTF-8 - -wds.linkis.server.mybatis.datasource.username= - -***REMOVED*** - - -wds.linkis.log.clear=true -wds.linkis.server.version=v1 - -##restful -wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.dss.server.restful,com.webank.wedatasphere.dss.application.restful - -##mybatis -wds.linkis.server.mybatis.mapperLocations=classpath:com/webank/wedatasphere/dss/server/dao/impl/*.xml,classpath*:com/webank/wedatasphere/dss/application/dao/impl/*.xml - -wds.linkis.server.mybatis.typeAliasesPackage=com.webank.wedatasphere.dss.server.entity,com.webank.wedatasphere.dss.application.entity - -wds.linkis.server.mybatis.BasePackage=com.webank.wedatasphere.dss.server.dao,com.webank.wedatasphere.dss.application.dao - -##azkaban -wds.dss.appjoint.scheduler.azkaban.address=http://0.0.0.0:8081 - -wds.linkis.gateway.ip=127.0.0.1 -wds.linkis.gateway.port=9001 - -wds.dss.appjoint.scheduler.project.store.dir=file:///appcom/tmp/wds/scheduler -wds.linkis.super.user.name=root -wds.linkis.workspace.user.root.path=file:///tmp/linkis/ -wds.linkis.hdfs.user.root.path=hdfs:///tmp/linkis -wds.linkis.result.set.root.path=hdfs:///tmp/linkis -wds.linkis.scheduler.path=file:///appcom/tmp/wds/scheduler -wds.linkis.user.path=hdfs:///user -wds.linkis.dss.install.dir=/usr/local/dss_linkis/dss/dss-server -wds.linkis.azkaban.install.dir=/usr/local/dss_linkis/azkaban - -wds.linkis.metastore.hive.hdfs.base.path=/user/hive/warehouse -wds.linkis.metastore.script.path=default/Metastore.sh -wds.linkis.metastore.db.tail=_default - -wds.linkis.kerberos.realm= -wds.linkis.kerberos.admin= -wds.linkis.kerberos.enable.switch=0 -wds.linkis.kerberos.script.path=default/Kerberos.sh -wds.linkis.kerberos.keytab.path=/etc/security/keytabs -wds.linkis.kerberos.kdc.node= -wds.linkis.kerberos.kdc.user.name= -wds.linkis.kerberos.kdc.user.password= -wds.linkis.kerberos.ssh.port=22 -wds.dss.deploy.path=/usr/local/dss_linkis -wds.dss.user.account.command.class=com.webank.wedatasphpere.dss.user.service.impl.LinuxUserCommand,com.webank.wedatasphpere.dss.user.service.impl.KerberosCommand,com.webank.wedatasphpere.dss.user.service.impl.LdapCommand,com.webank.wedatasphpere.dss.user.service.impl.WorkspaceCommand,com.webank.wedatasphpere.dss.user.service.impl.MetastoreCommand,com.webank.wedatasphpere.dss.user.service.impl.AzkabanCommand -wds.dss.scheduler.url=/schedule/system \ No newline at end of file diff --git a/dss-server/src/main/resources/log4j.properties b/dss-server/src/main/resources/log4j.properties deleted file mode 100644 index 0807e6087..000000000 --- a/dss-server/src/main/resources/log4j.properties +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO diff --git a/dss-server/src/main/resources/log4j2.xml b/dss-server/src/main/resources/log4j2.xml deleted file mode 100644 index 3923cd9f3..000000000 --- a/dss-server/src/main/resources/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/dss-server/src/main/resources/token.properties b/dss-server/src/main/resources/token.properties deleted file mode 100644 index 14963db72..000000000 --- a/dss-server/src/main/resources/token.properties +++ /dev/null @@ -1,2 +0,0 @@ -#${userName}=${password} -hadoop=hadoop diff --git a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/crumb/CrumbFactory.scala b/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/crumb/CrumbFactory.scala deleted file mode 100644 index 35df203a7..000000000 --- a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/crumb/CrumbFactory.scala +++ /dev/null @@ -1,144 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.crumb - -import com.webank.wedatasphere.dss.server.entity.CrumbType.CrumbType -import com.webank.wedatasphere.dss.server.entity.{Crumb, CrumbType, DSSFlowTaxonomy, DSSProjectTaxonomy} -import com.webank.wedatasphere.dss.server.service.impl.{DSSFlowServiceImpl, DSSFlowTaxonomyServiceImpl, DSSProjectServiceImpl, DSSProjectTaxonomyServiceImpl} -import org.springframework.beans.factory.annotation.Autowired -import org.springframework.stereotype.Component - -import scala.collection.mutable.ArrayBuffer - - -@Component -class CrumbFactory { - @Autowired - private var projectService: DSSProjectServiceImpl = _ - @Autowired - private var projectTaxonomyService:DSSProjectTaxonomyServiceImpl = _ - @Autowired - private var flowService: DSSFlowServiceImpl = _ - @Autowired - private var flowTaxonomyService:DSSFlowTaxonomyServiceImpl = _ - - def createCrumbs(crumbType: CrumbType, params: Array[String]): Array[Crumb] = { - crumbType match { - case CrumbType.All => Array(createIndexCrumb, createAllCrumb) - case CrumbType.SortProject => Array(createIndexCrumb, createAllCrumb, createSortProjectCrumb(params)) - case CrumbType.Project => Array(createIndexCrumb, createAllCrumb, createSortProjectCrumb(subParams(params,Array(1))), - createProjectCrumb(params)) - case CrumbType.SortFlow => Array(createIndexCrumb, createAllCrumb, createSortProjectCrumb(subParams(params,Array(1))), - createProjectCrumb(subParams(params,Array(1,2,4))), createSortFlowCrumb(params)) - case CrumbType.Flow => Array(createIndexCrumb, createAllCrumb, createSortProjectCrumb(subParams(params,Array(1))), - createProjectCrumb(subParams(params,Array(1,2,5))), createSortFlowCrumb(subParams(params,Array(1,2,3,5))), - createFlowCrumb(params)) - } - } - - private def subParams(params: Array[String], index: Array[Int]): Array[String] = { - val subParams = new ArrayBuffer[String] - for (i <- 1 to params.length) { - if (index.contains(i)) subParams += params(i - 1) - } - subParams.toArray - } - - private def createSortProjectCrumb(params: Array[String]): Crumb = { - val paramsMap = QuerParamsParser.toMap(params) - val projectTaxonomy = projectTaxonomyService.getProjectTaxonomyByID(paramsMap.get("projectTaxonomyID").toLong) - new Crumb(CrumbType.SortProject, QuerParamsParser.toQueryString(params), projectTaxonomy.getName, null) - } - - private def createProjectCrumb(params: Array[String]): Crumb = { - val paramsMap = QuerParamsParser.toMap(params) - val project = projectService.getProjectByProjectVersionID(paramsMap.get("projectVersionID").toLong) - new Crumb(CrumbType.Project, QuerParamsParser.toQueryString(params), project.getName, null) - } - - private def createSortFlowCrumb(params: Array[String]): Crumb = { - val paramsMap = QuerParamsParser.toMap(params) - val flowTaxonomy = flowTaxonomyService.getFlowTaxonomyByID(paramsMap.get("flowTaxonomyID").toLong) - new Crumb(CrumbType.SortFlow, QuerParamsParser.toQueryString(params), flowTaxonomy.getName, null) - } - - private def createFlowCrumb(params: Array[String]): Crumb = { - val paramsMap = QuerParamsParser.toMap(params) - val flow = flowService.getFlowByID(paramsMap.get("flowID").toLong) - new Crumb(CrumbType.Flow, QuerParamsParser.toQueryString(params), flow.getName, null) - } - - private def createIndexCrumb: Crumb = { - new Crumb(CrumbType.Index, "", "首页", null) - } - - private def createAllCrumb: Crumb = { - new Crumb(CrumbType.All, "", "工程开发", null) - } - - def createCrumbTree(crumbType: CrumbType, params: java.util.Map[String, String]): Crumb = { - crumbType match { - case CrumbType.All => - case CrumbType.SortProject => - case CrumbType.Project => - case CrumbType.SortFlow => - case CrumbType.Flow => - } - null - } - - def createCrumbData(crumbType: CrumbType, params: java.util.Map[String, String],userName:String): Any = { - crumbType match { - case CrumbType.All =>createAllData(userName, params.get("workspaceId").toLong) - case CrumbType.SortProject =>createSortProjectData(params.get("projectTaxonomyID").toLong,userName) - case CrumbType.Project =>createProjectData(params.get("projectVersionID").toLong,params.get("isRootFlow").toBoolean) - case CrumbType.SortFlow =>createSortFlowData(params.get("projectVersionID").toLong,params.get("flowTaxonomyID").toLong,params.get("isRootFlow").toBoolean) - case CrumbType.Flow =>null - } - } - - private def createAllData(userName:String, workspaceId: Long):java.util.List[DSSProjectTaxonomy] ={ - projectTaxonomyService.listAllProjectTaxonomy(userName, workspaceId) - } - - private def createSortProjectData(projectTaxonomyID:Long,userName:String):java.util.List[DSSProjectTaxonomy] ={ - projectTaxonomyService.listProjectTaxonomy(projectTaxonomyID,userName) - } - - private def createProjectData(projectVersionID:Long,isRootFlow:Boolean):java.util.List[DSSFlowTaxonomy] = { - flowTaxonomyService.listAllFlowTaxonomy(projectVersionID,isRootFlow) - } - - private def createSortFlowData(projectVersionID:Long,flowTaxonomyID:Long,isRootFlow:Boolean):java.util.List[DSSFlowTaxonomy] = { - flowTaxonomyService.listFlowTaxonomy(projectVersionID,flowTaxonomyID:Long,isRootFlow:Boolean) - } -} - -/*object Test { - def main(args: Array[String]): Unit = { - import collection.JavaConversions._ - val a = new util.HashMap[String, String]() - a.put("a", "b") - a += "c" -> "d" - a += "e" -> "f" - a += "g" -> "dh" - a += "cerq" -> "dwer" - val b = new CrumbFactory - print(b.removeParams(a, "565")) - } -}*/ diff --git a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/crumb/QuerParamsParser.scala b/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/crumb/QuerParamsParser.scala deleted file mode 100644 index 5b979f85f..000000000 --- a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/crumb/QuerParamsParser.scala +++ /dev/null @@ -1,80 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.crumb - - -import com.webank.wedatasphere.dss.server.entity.CrumbType -import com.webank.wedatasphere.dss.server.entity.CrumbType.CrumbType - - - -object QuerParamsParser { - - def getCrumbType(queryParams: String): CrumbType = { - queryParams.split("&").size match { - case 1 => CrumbType.All - case 2 => CrumbType.SortProject - case 4 => CrumbType.Project - case 5 => CrumbType.SortFlow - case 6 => CrumbType.Flow - } - } - - def toArray(queryParams: String): Array[String] = { - queryParams.split("&") - } - - def toMap(queryParams: Array[String]): java.util.Map[String, String] = { - import collection.JavaConverters._ - val regex = ("(.+)=(.+)").r - queryParams.map { - _ match { - case regex(k, v) => k -> v - } - }.toMap.asJava - } - - @Deprecated - def toQueryString(params: java.util.Map[String, String]): String = { - import collection.JavaConversions._ - val queryString = params.foldLeft("")((r, p) => s"${r}&${p._1}=${p._2}") - queryString.substring(1, queryString.length) - } - - - def toQueryString(params:Array[String]):String ={ - val queryString = params.foldLeft("")((r,p) =>s"${r}&${p}") - queryString.substring(1, queryString.length) - } -} - - -/*object Test{ - def main(args: Array[String]): Unit = { - val a = new util.HashMap[String,String]() - a.put("a","b") - a.put("c","d") - a.put("e","f") - val b = QuerParamsParser.toQueryString(a) - print(b.substring(1,b.length)) - } -}*/ - - - - diff --git a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/entity/Crumb.scala b/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/entity/Crumb.scala deleted file mode 100644 index c222b1961..000000000 --- a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/entity/Crumb.scala +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.entity - -import CrumbType.CrumbType - -import scala.beans.BeanProperty - - - -class Crumb(var crumbTypeE: CrumbType, @BeanProperty var params: String, @BeanProperty var name: String, @BeanProperty var childrens: List[Crumb]) { - @BeanProperty var crumbType: String = crumbTypeE.toString -} diff --git a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/entity/CrumbType.scala b/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/entity/CrumbType.scala deleted file mode 100644 index 957a8471d..000000000 --- a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/entity/CrumbType.scala +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.entity - - -object CrumbType extends Enumeration{ - type CrumbType = Value - val Index,All,SortProject,Project,SortFlow,Flow = Value -} diff --git a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/receiver/DSSServerReceiver.scala b/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/receiver/DSSServerReceiver.scala deleted file mode 100644 index 951128083..000000000 --- a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/receiver/DSSServerReceiver.scala +++ /dev/null @@ -1,49 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.receiver - -import com.webank.wedatasphere.dss.application.dao.ApplicationMapper -import com.webank.wedatasphere.dss.common.exception.DSSErrorException -import com.webank.wedatasphere.dss.common.protocol.{RequestDSSApplication, RequestDSSProject} -import com.webank.wedatasphere.dss.server.service.DSSProjectService -import com.webank.wedatasphere.linkis.rpc.{Receiver, Sender} -import org.springframework.beans.factory.annotation.Autowired -import org.springframework.stereotype.Component - -import scala.concurrent.duration.Duration - - -@Component -class DSSServerReceiver extends Receiver{ - - @Autowired - var dwsProjectService:DSSProjectService = _ - - @Autowired - var applicationMapper:ApplicationMapper = _ - - override def receive(message: Any, sender: Sender): Unit = {} - - override def receiveAndReply(message: Any, sender: Sender): Any = message match { - case f:RequestDSSProject => dwsProjectService.getExecutionDSSProject(f) - case RequestDSSApplication(name) => applicationMapper.getApplication(name) - case _ =>throw new DSSErrorException(90000,"") - } - - override def receiveAndReply(message: Any, duration: Duration, sender: Sender): Any = {} -} diff --git a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/service/BMLService.scala b/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/service/BMLService.scala deleted file mode 100644 index 8e2d380d2..000000000 --- a/dss-server/src/main/scala/com/webank/wedatasphere/dss/server/service/BMLService.scala +++ /dev/null @@ -1,111 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.server.service - -import java.io.{ByteArrayInputStream, InputStream} -import java.util -import java.util.UUID - -import com.webank.wedatasphere.dss.common.exception.DSSErrorException -import com.webank.wedatasphere.linkis.bml.client.{BmlClient, BmlClientFactory} -import com.webank.wedatasphere.linkis.bml.protocol.{BmlDownloadResponse, BmlUpdateResponse, BmlUploadResponse} -import org.springframework.stereotype.Component - -import scala.collection.JavaConversions._ - - -@Component -class BMLService { - - def upload(userName: String, content: String, fileName: String): util.Map[String, Object] = { - val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) - val client: BmlClient = createBMLClient(userName) - val resource: BmlUploadResponse = client.uploadResource(userName, fileName, inputStream) - if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") - val map = new util.HashMap[String, Object] - map += "resourceId" -> resource.resourceId - map += "version" -> resource.version - } - - def upload(userName: String, inputStream: InputStream, fileName: String): util.Map[String, Object] = { - val client: BmlClient = createBMLClient(userName) - val resource: BmlUploadResponse = client.uploadResource(userName, fileName, inputStream) - if (!resource.isSuccess) throw new DSSErrorException(911113, "上传失败") - val map = new util.HashMap[String, Object] - map += "resourceId" -> resource.resourceId - map += "version" -> resource.version - } - - def update(userName: String, resourceId: String, inputStream: InputStream): util.Map[String, Object] = { - val client: BmlClient = createBMLClient(userName) - val resource: BmlUpdateResponse = client.updateResource(userName, resourceId, "", inputStream) - if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") - val map = new util.HashMap[String, Object] - map += "resourceId" -> resource.resourceId - map += "version" -> resource.version - } - - def update(userName: String, resourceId: String, content: String): util.Map[String, Object] = { - val inputStream = new ByteArrayInputStream(content.getBytes("utf-8")) - val client: BmlClient = createBMLClient(userName) - val resource: BmlUpdateResponse = client.updateResource(userName, resourceId, UUID.randomUUID().toString+".json", inputStream) - if (!resource.isSuccess) throw new DSSErrorException(911114, "更新失败") - val map = new util.HashMap[String, Object] - map += "resourceId" -> resource.resourceId - map += "version" -> resource.version - } - - def query(userName: String, resourceId: String, version: String): util.Map[String, Object] = { - val client: BmlClient = createBMLClient(userName) - var resource: BmlDownloadResponse = null - if (version == null) { - resource = client.downloadResource(userName, resourceId) - } else { - resource = client.downloadResource(userName, resourceId, version) - } - if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") - val map = new util.HashMap[String, Object] - map += "path" -> resource.fullFilePath - map += "string" -> inputstremToString(resource.inputStream) - } - - def download(userName: String, resourceId: String, version: String): util.Map[String, Object] = { - val client: BmlClient = createBMLClient(userName) - var resource: BmlDownloadResponse = null - if (version == null) { - resource = client.downloadResource(userName, resourceId) - } else { - resource = client.downloadResource(userName, resourceId, version) - } - if (!resource.isSuccess) throw new DSSErrorException(911115, "下载失败") - val map = new util.HashMap[String, Object] - map += "path" -> resource.fullFilePath - map += "is" -> resource.inputStream - } - - private def inputstremToString(inputStream: InputStream): String = { - scala.io.Source.fromInputStream(inputStream).mkString - } - - private def createBMLClient(userName: String): BmlClient = { - if (userName == null) - BmlClientFactory.createBmlClient() - else - BmlClientFactory.createBmlClient(userName) - } -} diff --git a/dss-server/src/main/test/HttpTest.java b/dss-server/src/main/test/HttpTest.java deleted file mode 100644 index 0f2c6ac5f..000000000 --- a/dss-server/src/main/test/HttpTest.java +++ /dev/null @@ -1,185 +0,0 @@ -import com.webank.wedatasphere.dss.schedulis.exception.SchedulisSchedulerException; -import com.webank.wedatasphere.dss.schedulis.exception.SchedulisServerException; - -import org.apache.commons.io.IOUtils; -import org.apache.commons.lang.StringUtils; -import org.apache.http.Header; -import org.apache.http.HttpEntity; -import org.apache.http.NameValuePair; -import org.apache.http.client.CookieStore; -import org.apache.http.client.entity.UrlEncodedFormEntity; -import org.apache.http.client.methods.CloseableHttpResponse; -import org.apache.http.client.methods.HttpGet; -import org.apache.http.client.methods.HttpPost; -import org.apache.http.client.protocol.HttpClientContext; -import org.apache.http.cookie.Cookie; -import org.apache.http.entity.mime.MultipartEntityBuilder; -import org.apache.http.impl.client.BasicCookieStore; -import org.apache.http.impl.client.CloseableHttpClient; -import org.apache.http.impl.client.HttpClients; -import org.apache.http.message.BasicNameValuePair; -import org.apache.http.protocol.HTTP; -import org.apache.http.util.EntityUtils; -import org.junit.Test; -import scala.Tuple2; - -import java.io.*; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; -import java.util.Optional; - - -public class HttpTest { - - //@Test - public Cookie test01() throws IOException { - HttpPost httpPost = new HttpPost("http://127.0..0.1:8290/checkin"); - List params = new ArrayList<>(); - params.add(new BasicNameValuePair("username", "allenlliu")); - params.add(new BasicNameValuePair("userpwd", "*****")); - params.add(new BasicNameValuePair("action", "login")); - httpPost.setEntity(new UrlEncodedFormEntity(params)); - CookieStore cookieStore = new BasicCookieStore(); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - HttpClientContext context = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - context = HttpClientContext.create(); - response = httpClient.execute(httpPost, context); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - List cookies = context.getCookieStore().getCookies(); - return cookies.get(0); - } - - @Test - public void test02() throws IOException { - List strings = new ArrayList<>(); - strings.add("1"); - strings.add("bbb"); - strings.add("ccc"); - } - - public void print(String string) throws IOException, IllegalAccessException{ - switch (string){ - case "1":throw new IOException("a"); - case "2":throw new IllegalAccessException("b"); - default: - System.out.println(string); - } - } - @Test - public void test03() throws IOException, SchedulisSchedulerException { - Cookie cookie = test01(); - List params = new ArrayList<>(); - params.add(new BasicNameValuePair("ajax","fetchProjectPage")); - params.add(new BasicNameValuePair("start","0")); - params.add(new BasicNameValuePair("length","10")); - params.add(new BasicNameValuePair("projectsType","personal")); - params.add(new BasicNameValuePair("pageNum","1")); - params.add(new BasicNameValuePair("order","orderProjectName")); - CookieStore cookieStore = new BasicCookieStore(); - cookieStore.addCookie(cookie); - HttpClientContext context = HttpClientContext.create(); - CloseableHttpResponse response = null; - CloseableHttpClient httpClient = null; - try { - String finalUrl = "http://127.0.0.1:8088/index" + "?" + EntityUtils.toString(new UrlEncodedFormEntity(params)); - HttpGet httpGet = new HttpGet(finalUrl); - httpGet.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpGet, context); - /*Header[] allHeaders = context.getRequest().getAllHeaders(); - Optional
header = Arrays.stream(allHeaders).filter(f -> "Cookie".equals(f.getAppJointName())).findFirst(); - header.ifPresent(AzkabanUtils.handlingConsumerWrapper(this::parseCookie));*/ - } catch (Exception e) { - throw new SchedulisSchedulerException(90004, e.getMessage()); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - } - - @Test - public void test04(){ -/* A a = new A(); - Tuple2 hello = a.hello(); - Tuple2 stringStringTuple2 = new Tuple2<>("","");*/ - } - - @Test - public void test05(){ - String subFlowPath = "/Users/v_wbjftang/linuxDownloads/project_0926_tjf/flow2/subFlows/subFlow21/subFlows/subFlow211"; - int indexOf = subFlowPath.indexOf("subFlows"); - if(indexOf != -1){ - subFlowPath = subFlowPath.substring(0, indexOf-1); - } - String substring1 = subFlowPath.substring(0, subFlowPath.lastIndexOf("/")); - System.out.println(substring1); - String allPath = "/Users/v_wbjftang/linuxDownloads/project_0926_tjf/flow2/subFlows/subFlow21/subFlows/subFlow211"; - String substring = allPath.substring(substring1.length() +1 ); - System.out.println("\\" + File.separator); - //String s = substring.replaceAll("\\\\subFlows\\\\", "...."); - String s = substring.replaceAll("\\" + File.separator +"subFlows" + "\\" + File.separator, "...."); - System.out.println(s); - } - @Test - public void test06(){ - String nodeType = "spark.sql"; - String[] split = nodeType.split("\\."); - for (int i = 1; i < split.length; i++) { - String s = split[i]; - String replace = s.replace(s.charAt(0), (char) (s.charAt(0) - 32)); - split[i] = replace; - } - System.out.println(this.getClass().getPackage().getName()); - for (int i = 0; i < split.length; i++) { - System.out.println(i); - } - } - - @Test - public void test07() throws IOException { - Cookie cookie = test01(); - String projectName = "create_project0930"; - HttpPost httpPost = new HttpPost("http://127.0..0.1:8290/manager" + "?project=" + projectName); - httpPost.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - CloseableHttpResponse response = null; - File file = new File("E:\\appcom\\tmp\\dws\\wtss\\neiljianliu\\2019-09-30\\project_0926_tjf.zip"); - CookieStore cookieStore = new BasicCookieStore(); - cookieStore.addCookie(cookie); - CloseableHttpClient httpClient = null; - InputStream inputStream = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - MultipartEntityBuilder entityBuilder = MultipartEntityBuilder.create(); - entityBuilder.addBinaryBody("file",file); - entityBuilder.addTextBody("ajax", "upload"); - entityBuilder.addTextBody("project", projectName); - httpPost.setEntity(entityBuilder.build()); - response = httpClient.execute(httpPost); - HttpEntity httpEntity = response.getEntity(); - inputStream = httpEntity.getContent(); - String entStr = null; - entStr = IOUtils.toString(inputStream, "utf-8"); - if(response.getStatusLine().getStatusCode() != 200){ - throw new SchedulisServerException(90005, "release project failed, " + entStr); - } - }catch (Exception e){ - System.out.println(e.getMessage()); - } - finally { - IOUtils.closeQuietly(inputStream); - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - } - @Test - public void test08(){ - - } -} diff --git a/dss-server/src/main/test/com/webank/Adf.scala b/dss-server/src/main/test/com/webank/Adf.scala deleted file mode 100644 index a102f4147..000000000 --- a/dss-server/src/main/test/com/webank/Adf.scala +++ /dev/null @@ -1,9 +0,0 @@ -package com.webank - - -object Adf { - def main(args: Array[String]): Unit = { - val s = "10asdfasd" - print(s.stripPrefix("http://")) - } -} diff --git a/dss-server/src/main/test/com/webank/HttpTest.java b/dss-server/src/main/test/com/webank/HttpTest.java deleted file mode 100644 index 888ce006a..000000000 --- a/dss-server/src/main/test/com/webank/HttpTest.java +++ /dev/null @@ -1,141 +0,0 @@ -package com.webank; - -import com.webank.wedatasphere.dss.schedulis.exception.SchedulisSchedulerException; -import com.webank.wedatasphere.dss.schedulis.linkisjob.AbstractLinkisAzkabanJob; -import com.webank.wedatasphere.dss.schedulis.linkisjob.LinkisAzkabanJobFactory; -import org.apache.commons.io.IOUtils; -import org.apache.http.NameValuePair; -import org.apache.http.client.CookieStore; -import org.apache.http.client.entity.UrlEncodedFormEntity; -import org.apache.http.client.methods.CloseableHttpResponse; -import org.apache.http.client.methods.HttpGet; -import org.apache.http.client.methods.HttpPost; -import org.apache.http.client.protocol.HttpClientContext; -import org.apache.http.cookie.Cookie; -import org.apache.http.impl.client.BasicCookieStore; -import org.apache.http.impl.client.CloseableHttpClient; -import org.apache.http.impl.client.HttpClients; -import org.apache.http.message.BasicNameValuePair; -import org.apache.http.protocol.HTTP; -import org.apache.http.util.EntityUtils; -import org.junit.Test; - -import java.io.File; -import java.io.IOException; -import java.util.ArrayList; -import java.util.List; - - -public class HttpTest { - - //@Test - public Cookie test01() throws IOException { - HttpPost httpPost = new HttpPost("http://127.0.0.1:8088/checkin"); - List params = new ArrayList<>(); - params.add(new BasicNameValuePair("username", "neiljianliu")); - params.add(new BasicNameValuePair("userpwd", "*****")); - params.add(new BasicNameValuePair("action", "login")); - httpPost.setEntity(new UrlEncodedFormEntity(params)); - CookieStore cookieStore = new BasicCookieStore(); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - HttpClientContext context = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - context = HttpClientContext.create(); - response = httpClient.execute(httpPost, context); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - List cookies = context.getCookieStore().getCookies(); - return cookies.get(0); - } - - @Test - public void test02() throws IOException { - List strings = new ArrayList<>(); - strings.add("1"); - strings.add("bbb"); - strings.add("ccc"); - } - - public void print(String string) throws IOException, IllegalAccessException{ - switch (string){ - case "1":throw new IOException("a"); - case "2":throw new IllegalAccessException("b"); - default: - System.out.println(string); - } - } - @Test - public void test03() throws IOException, SchedulisSchedulerException { - Cookie cookie = test01(); - List params = new ArrayList<>(); - params.add(new BasicNameValuePair("ajax","fetchProjectPage")); - params.add(new BasicNameValuePair("start","0")); - params.add(new BasicNameValuePair("length","10")); - params.add(new BasicNameValuePair("projectsType","personal")); - params.add(new BasicNameValuePair("pageNum","1")); - params.add(new BasicNameValuePair("order","orderProjectName")); - CookieStore cookieStore = new BasicCookieStore(); - cookieStore.addCookie(cookie); - HttpClientContext context = HttpClientContext.create(); - CloseableHttpResponse response = null; - CloseableHttpClient httpClient = null; - try { - String finalUrl = "http://127.0.0.1:8088/index" + "?" + EntityUtils.toString(new UrlEncodedFormEntity(params)); - HttpGet httpGet = new HttpGet(finalUrl); - httpGet.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpGet, context); - /*Header[] allHeaders = context.getRequest().getAllHeaders(); - Optional
header = Arrays.stream(allHeaders).filter(f -> "Cookie".equals(f.getAppJointName())).findFirst(); - header.ifPresent(AzkabanUtils.handlingConsumerWrapper(this::parseCookie));*/ - } catch (Exception e) { - throw new SchedulisSchedulerException(90002, e.getMessage()); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - } - - @Test - public void test04(){ -/* A a = new A(); - Tuple2 hello = a.hello(); - Tuple2 stringStringTuple2 = new Tuple2<>("","");*/ - } - - @Test - public void test05(){ - //String subFlowPath = "C:\\Users\\v_wbjftang\\linuxDownloads\\project_0926_tjf\\flow2\\subFlows\\subFlow21\\subFlows\\subFlow211"; - String subFlowPath = "/Users/v_wbjftang/linuxDownloads/project_0926_tjf/flow2/subFlows/subFlow21/subFlows/subFlow211"; - int indexOf = subFlowPath.indexOf("subFlows"); - if(indexOf != -1){ - subFlowPath = subFlowPath.substring(0, indexOf-1); - } - String substring1 = subFlowPath.substring(0, subFlowPath.lastIndexOf("/")); - System.out.println(substring1); - String allPath = "/Users/v_wbjftang/linuxDownloads/project_0926_tjf/flow2/subFlows/subFlow21/subFlows/subFlow211"; - String substring = allPath.substring(substring1.length() +1 ); - System.out.println("\\" + File.separator); - //String s = substring.replaceAll("\\\\subFlows\\\\", "...."); - String s = substring.replaceAll("\\" + File.separator +"subFlows" + "\\" + File.separator, "...."); - System.out.println(s); - } - @Test - public void test06(){ -/* String nodeType = "spark.sql"; - try { - AbstractLinkisAzkabanJob job = LinkisAzkabanJobFactory.createJob(nodeType); - } catch (ClassNotFoundException e) { - e.printStackTrace(); - } catch (IllegalAccessException e) { - e.printStackTrace(); - } catch (InstantiationException e) { - e.printStackTrace(); - } - System.out.println("sdf");*/ - } -} diff --git a/dss-server/src/main/test/com/webank/JobParam.java b/dss-server/src/main/test/com/webank/JobParam.java deleted file mode 100644 index 482b060a3..000000000 --- a/dss-server/src/main/test/com/webank/JobParam.java +++ /dev/null @@ -1,11 +0,0 @@ -package com.webank; - - -public enum JobParam { - driver_memory; - - @Override - public String toString() { - return super.toString(); - } -} diff --git a/dss-server/src/main/test/com/webank/ReqTest.scala b/dss-server/src/main/test/com/webank/ReqTest.scala deleted file mode 100644 index 792e5ab96..000000000 --- a/dss-server/src/main/test/com/webank/ReqTest.scala +++ /dev/null @@ -1,9 +0,0 @@ -package com.webank - -import dispatch.Req - - -object ReqTest extends App{ - private val url: Req = dispatch.url("") - print(url == url.setMethod("POST")) -} diff --git a/dss-server/src/main/test/com/webank/TestUnit.java b/dss-server/src/main/test/com/webank/TestUnit.java deleted file mode 100644 index 07c619ae9..000000000 --- a/dss-server/src/main/test/com/webank/TestUnit.java +++ /dev/null @@ -1,14 +0,0 @@ -package com.webank; - -import com.webank.wedatasphpere.dss.user.service.impl.UserAuthorizationClient; -import org.junit.Test; - -public class TestUnit { - - @Test - public void test() throws Exception { - - UserAuthorizationClient UserAuthorizationClient = new UserAuthorizationClient(); - - } -} diff --git a/dss-standard/development-standard/development-process-standard-execution/pom.xml b/dss-standard/development-standard/development-process-standard-execution/pom.xml new file mode 100644 index 000000000..be32eef57 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/pom.xml @@ -0,0 +1,76 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-development-process-standard-execution + + + + com.webank.wedatasphere.dss + dss-development-process-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + net.alchim31.maven + scala-maven-plugin + + + + + \ No newline at end of file diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionListener.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionListener.java new file mode 100644 index 000000000..3614f0f08 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionListener.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.async; + + +public interface RefExecutionListener { +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionResponseListener.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionResponseListener.java new file mode 100644 index 000000000..8d3e5caa2 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionResponseListener.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.async; + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; + + +public interface RefExecutionResponseListener extends RefExecutionListener { + + void onRefExecutionCompleted(CompletedExecutionResponseRef response); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionStatusListener.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionStatusListener.java new file mode 100644 index 000000000..a3ed57296 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/async/RefExecutionStatusListener.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.async; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; + + +public interface RefExecutionStatusListener extends RefExecutionListener { + + void beforeSubmit(ExecutionRequestRef requestRef); + + void afterSubmit(ExecutionRequestRef requestRef, RefExecutionAction action); + + void afterAsyncResponseRef(AsyncExecutionResponseRef response); + + void afterCompletedExecutionResponseRef(ExecutionRequestRef requestRef, RefExecutionAction action, + CompletedExecutionResponseRef response); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AbstractRefExecutionAction.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AbstractRefExecutionAction.java new file mode 100644 index 000000000..e8b4fcf3b --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AbstractRefExecutionAction.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext; + + +public abstract class AbstractRefExecutionAction implements RefExecutionAction { + + private String id; + private ExecutionRequestRefContext executionRequestRefContext; + + public String getId() { + return id; + } + + public void setId(String id) { + this.id = id; + } + + public boolean isKilledFlag() { + return killedFlag; + } + + public void setKilledFlag(boolean killedFlag) { + this.killedFlag = killedFlag; + } + + private boolean killedFlag = false; + + @Override + public ExecutionRequestRefContext getExecutionRequestRefContext() { + return executionRequestRefContext; + } + + public void setExecutionRequestRefContext(ExecutionRequestRefContext executionRequestRefContext) { + this.executionRequestRefContext = executionRequestRefContext; + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncExecutionRequestRef.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncExecutionRequestRef.java new file mode 100644 index 000000000..dac40a9d4 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncExecutionRequestRef.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext; +import java.util.Map; + + +public interface AsyncExecutionRequestRef extends ExecutionRequestRef { + + ExecutionRequestRefContext getExecutionRequestRefContext(); + + void setExecutionRequestRefContext(ExecutionRequestRefContext executionRequestRefContext); + + void setProjectName(String projectName); + + void setOrchestratorName(String orchestratorName); + + void setJobContent(Map jobContent); + + void setName(String name); + + void setType(String type); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncExecutionResponseRef.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncExecutionResponseRef.java new file mode 100644 index 000000000..747451779 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncExecutionResponseRef.java @@ -0,0 +1,55 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.async.RefExecutionResponseListener; +import com.webank.wedatasphere.dss.standard.app.development.listener.core.LongTermRefExecutionOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public interface AsyncExecutionResponseRef extends ResponseRef { + + RefExecutionAction getAction(); + + void setAction(RefExecutionAction action); + + void addListener(RefExecutionResponseListener listener); + + void setExecutionRequestRef(ExecutionRequestRef requestRef); + + ExecutionRequestRef getExecutionRequestRef(); + + LongTermRefExecutionOperation getRefExecution(); + + void setRefExecution(LongTermRefExecutionOperation refExecution); + + long getAskStatePeriod(); + + void setAskStatePeriod(long askStatePeriod); + + void setCompleted(CompletedExecutionResponseRef response); + + boolean isCompleted(); + + long getStartTime(); + + long getMaxLoopTime(); + + void setMaxLoopTime(long maxLoopTime); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncResponseRefImpl.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncResponseRefImpl.java new file mode 100644 index 000000000..cfdc79fb3 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/AsyncResponseRefImpl.java @@ -0,0 +1,167 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.async.RefExecutionResponseListener; +import com.webank.wedatasphere.dss.standard.app.development.listener.core.LongTermRefExecutionOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractResponseRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AsyncResponseRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.function.Consumer; + + +public class AsyncResponseRefImpl extends AbstractResponseRef implements AsyncResponseRef, AsyncExecutionResponseRef { + + private RefExecutionAction action; + private ExecutionRequestRef requestRef; + private LongTermRefExecutionOperation refExecution; + private boolean isCompleted = false; + private CompletedExecutionResponseRef response; + private long startTime = System.currentTimeMillis(); + private long askStatePeriod = 1000; + private long maxLoopTime = -1; + private final Object lock = new Object(); + + /** + * 通过监听的方式，任务一旦完成，我们就通知上层 + */ + private List listeners = new ArrayList<>(); + + public AsyncResponseRefImpl(String responseBody, int status) { + super(responseBody, status); + } + + public AsyncResponseRefImpl() { + super(null, -1); + } + + @Override + public RefExecutionAction getAction() { + return action; + } + + @Override + public void setAction(RefExecutionAction action) { + this.action = action; + } + + @Override + public void addListener(RefExecutionResponseListener listener) { + this.listeners.add(listener); + } + + @Override + public void setExecutionRequestRef(ExecutionRequestRef requestRef) { + this.requestRef = requestRef; + } + + @Override + public ExecutionRequestRef getExecutionRequestRef() { + return requestRef; + } + + @Override + public LongTermRefExecutionOperation getRefExecution() { + return refExecution; + } + + @Override + public void setRefExecution(LongTermRefExecutionOperation refExecution) { + this.refExecution = refExecution; + } + + @Override + public long getAskStatePeriod() { + return askStatePeriod; + } + + @Override + public void setAskStatePeriod(long askStatePeriod) { + this.askStatePeriod = askStatePeriod; + } + + @Override + public boolean isCompleted() { + return isCompleted; + } + + @Override + public ResponseRef getResponse() { + return response; + } + + @Override + public void waitForCompleted() throws InterruptedException { + synchronized (lock) { + while(!isCompleted) { + lock.wait(2000); + } + } + } + + @Override + public void notifyMe(Consumer notifyListener) { + addListener(response -> notifyListener.accept(response)); + } + + @Override + public void setCompleted(CompletedExecutionResponseRef response) { + isCompleted = true; + this.response = response; + status = response.getStatus(); + responseBody = response.getResponseBody(); + synchronized (lock) { + lock.notifyAll(); + } + listeners.forEach(l -> l.onRefExecutionCompleted(response)); + } + + @Override + public long getMaxLoopTime() { + return maxLoopTime; + } + + @Override + public void setMaxLoopTime(long maxLoopTime) { + this.maxLoopTime = maxLoopTime; + } + + @Override + public long getStartTime() { + return startTime; + } + + @Override + public Map toMap() { + if(response != null) { + return response.toMap(); + } + return null; + } + + @Override + public String getErrorMsg() { + if(response != null) { + return response.getErrorMsg(); + } + return null; + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/CompletedExecutionResponseRef.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/CompletedExecutionResponseRef.java new file mode 100644 index 000000000..c40262e90 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/CompletedExecutionResponseRef.java @@ -0,0 +1,71 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractResponseRef; +import java.util.HashMap; +import java.util.Map; + + +public class CompletedExecutionResponseRef extends AbstractResponseRef { + + private Throwable exception; + + public CompletedExecutionResponseRef(int status) { + super(null, status); + } + + public CompletedExecutionResponseRef(String responseBody, int status) { + super(responseBody, status); + } + + public void setIsSucceed(boolean isSucceed) { + if(isSucceed) { + status = 200; + } else { + status = 400; + } + } + + public void setErrorMsg(String errorMsg) { + this.errorMsg = errorMsg; + } + + @Override + public String getErrorMsg() { + return errorMsg; + } + + public Throwable getException() { + return exception; + } + + public void setException(Throwable exception) { + this.exception = exception; + } + + @Override + public Map toMap() { + if(responseMap == null || (Integer) responseMap.get("status") == -1) { + responseMap = new HashMap<>(3); + responseMap.put("errorMsg", errorMsg); + responseMap.put("status", status); + responseMap.put("exception", exception); + } + return responseMap; + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/LongTermRefExecutionAction.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/LongTermRefExecutionAction.java new file mode 100644 index 000000000..2f8cc3824 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/LongTermRefExecutionAction.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + + +public interface LongTermRefExecutionAction extends RefExecutionAction { + + void setSchedulerId(int schedulerId); + + int getSchedulerId(); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/RefExecutionAction.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/RefExecutionAction.java new file mode 100644 index 000000000..b88677e0f --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/RefExecutionAction.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + +import com.webank.wedatasphere.dss.standard.app.development.listener.core.ExecutionRequestRefContext; + + +public interface RefExecutionAction { + + ExecutionRequestRefContext getExecutionRequestRefContext(); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/RefExecutionState.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/RefExecutionState.java new file mode 100644 index 000000000..27dc8eb49 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/common/RefExecutionState.java @@ -0,0 +1,67 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.common; + + +public enum RefExecutionState { + /** + * NodeExecution的状态枚举 + */ + Accepted(0, "Accepted"), + Running(1, "Running"), + Success(2, "Success"), + Failed(3, "Failed"), + Killed(4, "Killed"), + Alert(5, "Alert"); + + private int code; + private String status; + + private RefExecutionState(int code, String status){ + this.code = code; + this.status = status; + } + + public int getCode() { + return code; + } + + public String getStatus() { + return status; + } + + public static boolean isCompleted(RefExecutionState state){ + return Success.equals(state) || Failed.equals(state) || Killed.equals(state); + } + + public static boolean isCompleted(String state){ + return Success.status.equals(state) || + Failed.status.equals(state) || + Killed.status.equals(state); + } + + public boolean isCompleted(){ + return isCompleted(this); + } + + public boolean isSuccess(){ + return Success.equals(this); + } + + + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/CallbackLongTermRefExecutionOperation.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/CallbackLongTermRefExecutionOperation.java new file mode 100644 index 000000000..1812c59d8 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/CallbackLongTermRefExecutionOperation.java @@ -0,0 +1,53 @@ + +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; +import com.webank.wedatasphere.dss.standard.app.development.listener.conf.RefExecutionConfiguration; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + + +public abstract class CallbackLongTermRefExecutionOperation extends LongTermRefExecutionOperation { + + protected Map asyncResponses = new ConcurrentHashMap(); + + public String getCallbackURL(ExecutionRequestRefContext executionRequestRefContext){ + String urlSuffix = RefExecutionConfiguration.CALL_BACK_URL().getValue(); + return executionRequestRefContext.getGatewayUrl() + urlSuffix; + } + + public abstract void acceptCallback(Map callbackMap); + + protected void markCompleted(RefExecutionAction action, CompletedExecutionResponseRef resultResponse) { + AsyncExecutionResponseRef response = asyncResponses.get(action); + asyncResponses.remove(action); + response.setCompleted(resultResponse); + } + + @Override + protected AsyncExecutionResponseRef createAsyncResponseRef(ExecutionRequestRef requestRef, RefExecutionAction action) { + AsyncExecutionResponseRef response = super.createAsyncResponseRef(requestRef, action); + response.setAskStatePeriod(RefExecutionConfiguration.CALLBACK_REF_EXECUTION_REFRESH_INTERVAL().getValue().toLong()); + asyncResponses.put(action, response); + return response; + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/ExecutionRequestRefContext.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/ExecutionRequestRefContext.java new file mode 100644 index 000000000..e91e1626e --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/ExecutionRequestRefContext.java @@ -0,0 +1,96 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + +import com.webank.wedatasphere.linkis.common.io.FsPath; +import com.webank.wedatasphere.linkis.common.io.MetaData; +import com.webank.wedatasphere.linkis.common.io.Record; +import com.webank.wedatasphere.linkis.common.io.resultset.ResultSet; +import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetReader; +import com.webank.wedatasphere.linkis.common.io.resultset.ResultSetWriter; +import java.util.HashMap; +import java.util.Map; + + +public interface ExecutionRequestRefContext { + + + static final String TOKEN_USER_STR = "Token-User"; + + static final String TOKEN_CODE_STR = "Token-Code"; + + + public Map getRuntimeMap(); + + public void appendLog(String log); + + public void updateProgress(float progress); + + LinkisJob fetchLinkisJob(long jobId); + + //获取任务的所有的结果集路径 + FsPath[] fetchLinkisJobResultSetPaths(long jobId); + + /** + * 获取本节点的操作用户 + * @return + */ + public String getUser(); + + void setStorePath(String storePath); + + String getStorePath(); + + ResultSetWriter createTableResultSetWriter(); + + ResultSetWriter createTableResultSetWriter(String resultSetAlias); + + ResultSetWriter createTextResultSetWriter(); + + ResultSetWriter createTextResultSetWriter(String resultSetAlias); + + ResultSetWriter createHTMLResultSetWriter(); + + ResultSetWriter createHTMLResultSetWriter(String resultSetAlias); + + ResultSetWriter createPictureResultSetWriter(); + + ResultSetWriter createPictureResultSetWriter(String resultSetAlias); + + ResultSetWriter createResultSetWriter(ResultSet resultSet, + String resultSetAlias); + + + ResultSetReader getResultSetReader(FsPath fsPath); + + void sendResultSet(ResultSetWriter resultSetWriter); + + /** + * + * @return + */ + String getGatewayUrl(); + + + default Map getTokenHeader(String user){ + Map tokenHeader = new HashMap<>(); + tokenHeader.put(TOKEN_CODE_STR, "dss-AUTH"); + tokenHeader.put(TOKEN_USER_STR, user); + return tokenHeader; + } + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/HttpLongTermRefExecution.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/HttpLongTermRefExecution.java new file mode 100644 index 000000000..a64599e2d --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/HttpLongTermRefExecution.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + + +public abstract class HttpLongTermRefExecution extends LongTermRefExecutionOperation { + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/Killable.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/Killable.java new file mode 100644 index 000000000..4a871c2db --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/Killable.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; + + +public interface Killable { + public boolean kill(RefExecutionAction action); +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/LinkisJob.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/LinkisJob.java new file mode 100644 index 000000000..2455d63be --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/LinkisJob.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + +import com.webank.wedatasphere.linkis.manager.label.entity.Label; +import java.util.List; +import java.util.Map; + + +public interface LinkisJob { + + String getResultLocation(); + + String getSubmitUser(); + + String getExecuteUser(); + + String getStatus(); + + Map getSource(); + + Map getParams(); + + List> getLabels(); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/LongTermRefExecutionOperation.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/LongTermRefExecutionOperation.java new file mode 100644 index 000000000..a907c7ed0 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/LongTermRefExecutionOperation.java @@ -0,0 +1,113 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; +import com.webank.wedatasphere.dss.standard.app.development.listener.async.RefExecutionStatusListener; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AbstractRefExecutionAction; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncResponseRefImpl; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.CompletedExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionState; +import com.webank.wedatasphere.dss.standard.app.development.listener.exception.AppConnExecutionWarnException; +import com.webank.wedatasphere.dss.standard.app.development.listener.scheduler.LongTermRefExecutionScheduler; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import java.util.ArrayList; +import java.util.List; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Description: LongTermRefExecutionOperation is used to execute long-term tasks in external application systems. + * Long-term task usually supports to execute asyncly and can fetch status and logs when it is submitted. + * + */ +public abstract class LongTermRefExecutionOperation implements RefExecutionOperation { + + private static final Logger logger = LoggerFactory.getLogger(LongTermRefExecutionOperation.class); + + private List refExecutionListener = new ArrayList(); + private LongTermRefExecutionScheduler scheduler = SchedulerManager.getScheduler(); + + public void addRefExecutionStatusListener(RefExecutionStatusListener refExecutionListener) { + for(RefExecutionStatusListener listener : this.refExecutionListener){ + if (listener.getClass().equals(refExecutionListener.getClass())){ + return; + } + } + this.refExecutionListener.add(refExecutionListener); + } + + public LongTermRefExecutionScheduler getScheduler() { + return scheduler; + } + + public void setScheduler(LongTermRefExecutionScheduler scheduler) { + this.scheduler = scheduler; + } + + protected abstract RefExecutionAction submit(ExecutionRequestRef requestRef); + + public abstract RefExecutionState state(RefExecutionAction action); + + public abstract CompletedExecutionResponseRef result(RefExecutionAction action); + + + protected ExecutionRequestRefContext createExecutionRequestRefContext(ExecutionRequestRef requestRef) { + if(requestRef instanceof AsyncExecutionRequestRef) { + return ((AsyncExecutionRequestRef) requestRef).getExecutionRequestRefContext(); + } else { + throw new AppConnExecutionWarnException(75536, "Cannot find a available ExecutionRequestRefContext."); + } + } + + @Override + public ResponseRef execute(ExecutionRequestRef requestRef) { + refExecutionListener.forEach(l -> l.beforeSubmit(requestRef)); + RefExecutionAction action = submit(requestRef); + if(action instanceof AbstractRefExecutionAction) { + ((AbstractRefExecutionAction) action).setExecutionRequestRefContext(createExecutionRequestRefContext(requestRef)); + } + refExecutionListener.forEach(l -> l.afterSubmit(requestRef, action)); + RefExecutionState state = state(action); + if(state != null && state.isCompleted()) { + CompletedExecutionResponseRef response = result(action); + refExecutionListener.forEach(l -> l.afterCompletedExecutionResponseRef(requestRef, action, response)); + return response; + } else { + AsyncExecutionResponseRef response = createAsyncResponseRef(requestRef, action); + response.setRefExecution(this); + refExecutionListener.forEach(l -> l.afterAsyncResponseRef(response)); + response.addListener(r -> refExecutionListener.forEach(l -> l.afterCompletedExecutionResponseRef(requestRef, action, (CompletedExecutionResponseRef) r))); + if(scheduler != null) { + scheduler.addAsyncResponse(response); + } + return response; + } + } + + protected AsyncExecutionResponseRef createAsyncResponseRef(ExecutionRequestRef requestRef, RefExecutionAction action) { + AsyncResponseRefImpl response = new AsyncResponseRefImpl(); + response.setAction(action); + response.setExecutionRequestRef(requestRef); + return response; + } + +} diff --git a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/ParamsNode.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/ParamsNode.java similarity index 76% rename from dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/ParamsNode.java rename to dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/ParamsNode.java index ac8152ca0..ddd972b0a 100644 --- a/dss-appjoint-core/src/main/java/com/webank/wedatasphere/dss/appjoint/execution/core/ParamsNode.java +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/ParamsNode.java @@ -1,8 +1,7 @@ /* * Copyright 2019 WeBank - * * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. + * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 @@ -15,14 +14,11 @@ * */ -package com.webank.wedatasphere.dss.appjoint.execution.core; +package com.webank.wedatasphere.dss.standard.app.development.listener.core; import java.util.Map; -/** - * created by enjoyyin on 2019/9/29 - * Description: - */ + public interface ParamsNode { public Map getParams(); } diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/Procedure.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/Procedure.java new file mode 100644 index 000000000..53e21a08d --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/Procedure.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.RefExecutionAction; + + +public interface Procedure { + + public float progress(RefExecutionAction action); + + public String log(RefExecutionAction action); + + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/SchedulerManager.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/SchedulerManager.java new file mode 100644 index 000000000..9b96e3172 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/core/SchedulerManager.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.core; + +import com.webank.wedatasphere.dss.standard.app.development.listener.scheduler.ListenerEventBusRefExecutionScheduler; +import com.webank.wedatasphere.dss.standard.app.development.listener.scheduler.LongTermRefExecutionScheduler; + + +public class SchedulerManager { + + private static LongTermRefExecutionScheduler scheduler = new ListenerEventBusRefExecutionScheduler(); + + static{ + ((ListenerEventBusRefExecutionScheduler)scheduler).start(); + } + + public static LongTermRefExecutionScheduler getScheduler(){ + return scheduler; + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/exception/AppConnExecutionErrorException.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/exception/AppConnExecutionErrorException.java new file mode 100644 index 000000000..117641cc4 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/exception/AppConnExecutionErrorException.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.exception; + +import com.webank.wedatasphere.linkis.common.exception.ErrorException; + + +public class AppConnExecutionErrorException extends ErrorException { + + public AppConnExecutionErrorException(int errCode, String desc) { + super(errCode, desc); + } + + public AppConnExecutionErrorException(int errCode, String desc, Throwable cause) { + super(errCode, desc); + initCause(cause); + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/exception/AppConnExecutionWarnException.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/exception/AppConnExecutionWarnException.java new file mode 100644 index 000000000..7f95f6b11 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/exception/AppConnExecutionWarnException.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.exception; + +import com.webank.wedatasphere.linkis.common.exception.WarnException; + + +public class AppConnExecutionWarnException extends WarnException { + + public AppConnExecutionWarnException(int errCode, String desc) { + super(errCode, desc); + } + + public AppConnExecutionWarnException(int errCode, String desc, Throwable cause) { + super(errCode, desc); + initCause(cause); + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/AsyncRefExecutionSchedulerListener.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/AsyncRefExecutionSchedulerListener.java new file mode 100644 index 000000000..6e31d3c51 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/AsyncRefExecutionSchedulerListener.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.scheduler; + +import com.webank.wedatasphere.linkis.common.listener.Event; +import com.webank.wedatasphere.linkis.common.listener.EventListener; + + +public interface AsyncRefExecutionSchedulerListener extends EventListener { + + void onEvent(AsyncResponseRefEvent event); + + void onEventError(AsyncResponseRefEvent event, Throwable t); + + @Override + default void onEventError(Event event, Throwable t) { + if(event instanceof AsyncResponseRefEvent) { + onEventError((AsyncResponseRefEvent) event, t); + } + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/AsyncResponseRefEvent.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/AsyncResponseRefEvent.java new file mode 100644 index 000000000..6ba615bc2 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/AsyncResponseRefEvent.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.scheduler; + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionResponseRef; +import com.webank.wedatasphere.linkis.common.listener.Event; + + +public class AsyncResponseRefEvent implements Event { + + private AsyncExecutionResponseRef response; + private long lastAskTime = 0; + + public AsyncResponseRefEvent(AsyncExecutionResponseRef response) { + this.response = response; + } + + public AsyncExecutionResponseRef getResponse() { + return response; + } + + public void setResponse(AsyncExecutionResponseRef response) { + this.response = response; + } + + public long getLastAskTime() { + return lastAskTime; + } + + public void setLastAskTime() { + this.lastAskTime = System.currentTimeMillis(); + } +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/LongTermRefExecutionScheduler.java b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/LongTermRefExecutionScheduler.java new file mode 100644 index 000000000..b91fad1b6 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/LongTermRefExecutionScheduler.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.scheduler; + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.AsyncExecutionResponseRef; +import com.webank.wedatasphere.dss.standard.app.development.listener.common.LongTermRefExecutionAction; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AsyncResponseRef; + + +public interface LongTermRefExecutionScheduler { + + void addAsyncResponse(AsyncExecutionResponseRef response); + + void removeAsyncResponse(LongTermRefExecutionAction action); + + AsyncResponseRef getAsyncResponse(LongTermRefExecutionAction action); + + void start(); + + void stop(); + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/scala/com/webank/wedatasphere/dss/standard/app/development/listener/conf/RefExecutionConfiguration.scala b/dss-standard/development-standard/development-process-standard-execution/src/main/scala/com/webank/wedatasphere/dss/standard/app/development/listener/conf/RefExecutionConfiguration.scala new file mode 100644 index 000000000..ae5b34953 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/scala/com/webank/wedatasphere/dss/standard/app/development/listener/conf/RefExecutionConfiguration.scala @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.conf + +import com.webank.wedatasphere.linkis.common.conf.{CommonVars, TimeType} + + +object RefExecutionConfiguration { + + val JOB_HISTORY_APPLICATION_NAME = CommonVars("wds.dss.appconn.jobhistory.service.name", "linkis-ps-publicservice") + val CALL_BACK_URL = CommonVars("wds.dss.appconn.ref-execution.callback.url", "/api/rest_j/v1/engineconn/callback") + + val ASYNC_REF_EXECUTION_SCHEDULER_QUEUE_SIZE = CommonVars("wds.dss.appconn.ref-execution.scheduler.queue.size", 2000) + val ASYNC_REF_EXECUTION_SCHEDULER_THREAD_SIZE = CommonVars("wds.dss.appconn.ref-execution.scheduler.thread.max", 20) + + val CALLBACK_REF_EXECUTION_REFRESH_INTERVAL = CommonVars("wds.dss.appconn.ref-execution.callback.refresh.interval", new TimeType("2m")) + +} diff --git a/dss-standard/development-standard/development-process-standard-execution/src/main/scala/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/ListenerEventBusRefExecutionScheduler.scala b/dss-standard/development-standard/development-process-standard-execution/src/main/scala/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/ListenerEventBusRefExecutionScheduler.scala new file mode 100644 index 000000000..79215522a --- /dev/null +++ b/dss-standard/development-standard/development-process-standard-execution/src/main/scala/com/webank/wedatasphere/dss/standard/app/development/listener/scheduler/ListenerEventBusRefExecutionScheduler.scala @@ -0,0 +1,126 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener.scheduler + +import java.util +import java.util.concurrent.ArrayBlockingQueue + +import com.webank.wedatasphere.dss.standard.app.development.listener.common.{AbstractRefExecutionAction, AsyncExecutionResponseRef, AsyncResponseRefImpl, CompletedExecutionResponseRef, LongTermRefExecutionAction} +import com.webank.wedatasphere.dss.standard.app.development.listener.conf.RefExecutionConfiguration._ +import com.webank.wedatasphere.dss.standard.app.development.listener.exception.AppConnExecutionErrorException +import com.webank.wedatasphere.linkis.common.listener.ListenerEventBus +import com.webank.wedatasphere.linkis.common.utils.{ByteTimeUtils, Utils} +import org.apache.commons.lang.exception.ExceptionUtils +import org.apache.commons.lang.time.DateFormatUtils + + +class ListenerEventBusRefExecutionScheduler(eventQueueCapacity: Int, name: String)(listenerConsumerThreadSize: Int) + extends LongTermRefExecutionScheduler { + + private val listenerEventBus = new ListenerEventBus[AsyncRefExecutionSchedulerListener, AsyncResponseRefEvent](eventQueueCapacity, name)(listenerConsumerThreadSize) { + override protected val dropEvent: DropEvent = new DropEvent { + override def onDropEvent(event: AsyncResponseRefEvent): Unit = throw new AppConnExecutionErrorException(95536, "LongTermNodeExecutionScheduler is full, please ask admin for help!") + override def onBusStopped(event: AsyncResponseRefEvent): Unit = throw new AppConnExecutionErrorException(95536, "LongTermNodeExecutionScheduler is stopped, please ask admin for help!") + } + override def doPostEvent(listener: AsyncRefExecutionSchedulerListener, event: AsyncResponseRefEvent): Unit = { + listener.onEvent(event) + } + } + + def this() = { + this(ASYNC_REF_EXECUTION_SCHEDULER_QUEUE_SIZE.getValue, "Async-NodeExecution-Scheduler")(ASYNC_REF_EXECUTION_SCHEDULER_THREAD_SIZE.getValue) + getAsyncRefExecutionSchedulerListeners.foreach(listenerEventBus.addListener) + } + + private val eventQueue = { + val ru = scala.reflect.runtime.universe + val classMirror = ru.runtimeMirror(getClass.getClassLoader) + val listenerEventBusClass = classMirror.reflect(listenerEventBus) + val field1 = ru.typeOf[ListenerEventBus[_, _]].decl(ru.TermName("eventQueue")).asMethod + val result = listenerEventBusClass.reflectMethod(field1) + result() match { + case queue: ArrayBlockingQueue[AsyncResponseRefEvent] => queue + } + } + + protected def getAsyncRefExecutionSchedulerListeners: Array[AsyncRefExecutionSchedulerListener] = { + Array(new AsyncRefExecutionSchedulerListener() { + override def onEvent(event: AsyncResponseRefEvent): Unit = if(!event.getResponse.isCompleted) { + val response = event.getResponse + val action = response.getAction.asInstanceOf[AbstractRefExecutionAction] + if(action.isKilledFlag){ + val resultResponse = response.getRefExecution.result(response.getAction) + onEventCompleted(event, resultResponse) + } + if(response.getMaxLoopTime > 0 && System.currentTimeMillis - response.getStartTime >= response.getMaxLoopTime) { + onEventError(event, new AppConnExecutionErrorException(75533, s"AppConnNode Execution is overtime! StartTime is ${DateFormatUtils.format(response.getStartTime, "yyyy-MM-dd HH:mm:ss")}, maxWaitTime is " + + ByteTimeUtils.msDurationToString(response.getMaxLoopTime))) + return + } + val period = System.currentTimeMillis() - event.getLastAskTime + if(period < response.getAskStatePeriod) { + if(period < 10) Utils.sleepQuietly(100) + if(!response.isCompleted) addEvent(event) + return + } + val state = response.getRefExecution.state(response.getAction) + if(state.isCompleted) { + val resultResponse = response.getRefExecution.result(response.getAction) + onEventCompleted(event, resultResponse) + } else if(!response.isCompleted) { + event.setLastAskTime() + addEvent(event) + } + } + + private def onEventCompleted(event: AsyncResponseRefEvent, response: CompletedExecutionResponseRef): Unit = { + event.getResponse.setCompleted(response) + } + + override def onEventError(event: AsyncResponseRefEvent, t: Throwable): Unit = t match { + case e: Exception => + val response = new CompletedExecutionResponseRef(400) { + override def toMap: util.Map[String, AnyRef] = new util.HashMap[String, AnyRef] + override def getErrorMsg: String = ExceptionUtils.getRootCauseMessage(e) + } + onEventCompleted(event, response) + } + }) + } + + override def addAsyncResponse(response: AsyncExecutionResponseRef): Unit = + addEvent(new AsyncResponseRefEvent(response)) + + protected def addEvent(event: AsyncResponseRefEvent): Unit = synchronized { + listenerEventBus.post(event) +// event.getResponse.getAction match { +// case longTermAction: LongTermRefExecutionAction => +// longTermAction.setSchedulerId(eventQueue.max) +// case _ => +// } + } + + override def removeAsyncResponse(action: LongTermRefExecutionAction): Unit = { + + } + + override def getAsyncResponse(action: LongTermRefExecutionAction): AsyncResponseRefImpl = null + + override def start(): Unit = listenerEventBus.start() + + override def stop(): Unit = listenerEventBus.stop() +} \ No newline at end of file diff --git a/dss-standard/development-standard/development-process-standard/pom.xml b/dss-standard/development-standard/development-process-standard/pom.xml new file mode 100644 index 000000000..f62cfe43e --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/pom.xml @@ -0,0 +1,63 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-development-process-standard + + + + org.apache.maven.plugins + maven-compiler-plugin + + 8 + 8 + + + + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + + \ No newline at end of file diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/ExecutionLogListener.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/ExecutionLogListener.java new file mode 100644 index 000000000..9610b39ff --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/ExecutionLogListener.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener; + +public interface ExecutionLogListener { + + void onInfo(String log); + void onERROR(String log); + void onWarn(String log); + void onSystemInfo(String log); + void onSystemError(String log); + void onSystemWarn(String log); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/ExecutionResultListener.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/ExecutionResultListener.java new file mode 100644 index 000000000..7ce85faa0 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/listener/ExecutionResultListener.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.listener; + +import com.webank.wedatasphere.linkis.common.io.MetaData; +import com.webank.wedatasphere.linkis.common.io.Record; + +public interface ExecutionResultListener { + + void setResultSetType(String resultSetType); + void onResultMetaData(MetaData metaData); + void onResultSetRecord(Record record); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/DevelopmentOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/DevelopmentOperation.java new file mode 100644 index 000000000..77f0dc6c6 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/DevelopmentOperation.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + + + +import com.webank.wedatasphere.dss.standard.app.development.service.DevelopmentService; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.service.Operation; + +public interface DevelopmentOperation extends Operation { + + void setDevelopmentService(DevelopmentService service); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefCopyOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefCopyOperation.java new file mode 100644 index 000000000..b0a3ff95f --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefCopyOperation.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.app.development.ref.CopyRequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefCopyOperation + extends DevelopmentOperation { + + + ResponseRef copyRef(K requestRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefCreationOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefCreationOperation.java new file mode 100644 index 000000000..65e0d9e6c --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefCreationOperation.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.app.development.ref.CreateRequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefCreationOperation extends DevelopmentOperation { + + ResponseRef createRef(K requestRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefDeletionOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefDeletionOperation.java new file mode 100644 index 000000000..a2bb55072 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefDeletionOperation.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefDeletionOperation extends DevelopmentOperation { + + void deleteRef(K requestRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefExecutionOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefExecutionOperation.java new file mode 100644 index 000000000..aef3460be --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefExecutionOperation.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.app.development.ref.ExecutionRequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefExecutionOperation + extends DevelopmentOperation { + + ResponseRef execute(ExecutionRequestRef requestRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefExportOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefExportOperation.java new file mode 100644 index 000000000..1c78d7f8d --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefExportOperation.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefExportOperation extends DevelopmentOperation { + + ResponseRef exportRef(K requestRef) throws ExternalOperationFailedException; + +} + + diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefImportOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefImportOperation.java new file mode 100644 index 000000000..117206da0 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefImportOperation.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefImportOperation extends DevelopmentOperation { + + ResponseRef importRef(K requestRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefQueryOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefQueryOperation.java new file mode 100644 index 000000000..6bdfb4ca2 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefQueryOperation.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + +public interface RefQueryOperation + extends DevelopmentOperation { + + ResponseRef query(K requestRef) throws ExternalOperationFailedException; + + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefUpdateOperation.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefUpdateOperation.java new file mode 100644 index 000000000..a0911dc3a --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/operation/RefUpdateOperation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.operation; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + +public interface RefUpdateOperation + extends DevelopmentOperation { + + ResponseRef updateRef(K requestRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CommonRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CommonRequestRef.java new file mode 100644 index 000000000..15f0df642 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CommonRequestRef.java @@ -0,0 +1,62 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; + + +public interface CommonRequestRef extends WorkspaceRequestRef { + + void setUserName(String userName); + + String getUserName(); + + void setProjectId(Long projectId); + + Long getProjectId(); + + void setProjectName(String projectName); + + String getProjectName(); + + void setOrcName(String orcName); + + String getOrcName(); + + void setOrcId(Long orcId); + + Long getOrcId(); + + void setName(String name); + + void setWorkspaceName(String workspaceName); + + String getWorkspaceName(); + + + String getContextID(); + + + void setContextID(String contextIDStr); + + void setDSSLabels(List dssLabels); + + @Override + List getDSSLabels(); +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CommonResponseRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CommonResponseRef.java new file mode 100644 index 000000000..dde5771f0 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CommonResponseRef.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public class CommonResponseRef extends com.webank.wedatasphere.dss.standard.common.entity.ref.CommonResponseRef { + + protected Long orcId; + protected String content; + protected String name; + protected boolean result; + + public boolean getResult() { + return this.result; + } + + public void setResult(boolean result) { + this.result = result; + } + + public CommonResponseRef(String responseBody, int status) { + super(responseBody, status); + } + + public CommonResponseRef(){ + super("",0); + } + + public void setOrcId(Long orcId) { + this.orcId = orcId; + } + + public Long getOrcId() { + return orcId; + } + + public String getContent() { + return this.content; + } + + public void setContent(String content) { + this.content = content; + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CopyRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CopyRequestRef.java new file mode 100644 index 000000000..e98fdebbe --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CopyRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface CopyRequestRef extends WorkspaceRequestRef { +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CreateRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CreateRequestRef.java new file mode 100644 index 000000000..4f955c5ce --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/CreateRequestRef.java @@ -0,0 +1,23 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface CreateRequestRef extends WorkspaceRequestRef { + + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/DSSCommonResponseRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/DSSCommonResponseRef.java new file mode 100644 index 000000000..a1165d59e --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/DSSCommonResponseRef.java @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +import org.codehaus.jackson.JsonNode; +import org.codehaus.jackson.map.ObjectMapper; + +import java.text.SimpleDateFormat; +import java.util.Map; + + +public class DSSCommonResponseRef extends CommonResponseRef { + + private ObjectMapper jacksonJson = new ObjectMapper(); + + public DSSCommonResponseRef(String responseBody) throws Exception { + super(responseBody, 0); + if (responseMap.containsKey("status")) { + status = getInt(responseMap.get("status")); + if (status != 0 && status != 200) { + errorMsg = responseMap.get("message").toString(); + } + } else if (responseMap.containsKey("header")) { + Map headerMap = (Map) responseMap.get("header"); + if (headerMap.containsKey("code")) { + status = getInt(headerMap.get("code")); + if (status != 0 && status != 200) { + errorMsg = headerMap.get("msg").toString(); + } + } + } + } + + public static Integer getInt(Object original){ + if(original instanceof Double){ + return ((Double) original).intValue(); + } + return (Integer) original; + } +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/DeleteRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/DeleteRequestRef.java new file mode 100644 index 000000000..afb6511cf --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/DeleteRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface DeleteRequestRef extends WorkspaceRequestRef { +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ExecutionRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ExecutionRequestRef.java new file mode 100644 index 000000000..9aaefd80f --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ExecutionRequestRef.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +import java.util.Map; + +public interface ExecutionRequestRef extends WorkspaceRequestRef { + + /** + * 获取的node所属的project的id + * @return + */ + long getProjectId(); + + /** + * 获取node所属的project的name + * @return + */ + String getProjectName(); + String getOrchestratorName(); + + String getOrchestratorVersion(); + + long getOrchestratorId(); + + /** + * 获取到Node的执行内容,执行内容是以Map的形式的 + * @return + */ + Map getJobContent(); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ExportRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ExportRequestRef.java new file mode 100644 index 000000000..29cf93136 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ExportRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface ExportRequestRef extends WorkspaceRequestRef { +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ImportRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ImportRequestRef.java new file mode 100644 index 000000000..1eb7a1185 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/ImportRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface ImportRequestRef extends WorkspaceRequestRef { +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/NodeRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/NodeRequestRef.java new file mode 100644 index 000000000..7a3e19bf0 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/NodeRequestRef.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +import java.util.Map; + + +public interface NodeRequestRef extends CommonRequestRef, CreateRequestRef,UpdateRequestRef { + + void setNodeType(String nodeType); + + String getNodeType(); + + void setJobContent(Map jobContent); + + Map getJobContent(); + + +} \ No newline at end of file diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/OpenRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/OpenRequestRef.java new file mode 100644 index 000000000..83ac0db3f --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/OpenRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface OpenRequestRef extends WorkspaceRequestRef { +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UpdateCSRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UpdateCSRequestRef.java new file mode 100644 index 000000000..696127a09 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UpdateCSRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +public interface UpdateCSRequestRef extends NodeRequestRef { + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UpdateRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UpdateRequestRef.java new file mode 100644 index 000000000..a49629440 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UpdateRequestRef.java @@ -0,0 +1,21 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + + +public interface UpdateRequestRef extends WorkspaceRequestRef { +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UrlResponseRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UrlResponseRef.java new file mode 100644 index 000000000..3bdbbab70 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/UrlResponseRef.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public interface UrlResponseRef extends ResponseRef { + + String getUrl(); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/WorkspaceRequestRef.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/WorkspaceRequestRef.java new file mode 100644 index 000000000..ff0500604 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/WorkspaceRequestRef.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref; + +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; + +public interface WorkspaceRequestRef extends RequestRef { + + default void setWorkspace(Workspace workspace){ + + } + + default Workspace getWorkspace(){ + return null; + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/impl/CommonRequestRefImpl.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/impl/CommonRequestRefImpl.java new file mode 100644 index 000000000..1088584d2 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/impl/CommonRequestRefImpl.java @@ -0,0 +1,131 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref.impl; + +import com.google.common.collect.Maps; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.app.development.ref.CommonRequestRef; +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractRequestRef; + +import java.util.List; +import java.util.Map; + + +public class CommonRequestRefImpl extends AbstractRequestRef implements CommonRequestRef { + + protected String userName; + protected Long projectId; + protected String projectName; + protected String orcName; + protected Long orcId; + protected Workspace workspace; + protected String workspaceName; + protected List dssLabelList; + protected String contextID; + + @Override + public String getUserName() { + return userName; + } + + @Override + public void setUserName(String username) { + this.userName = username; + } + + @Override + public void setProjectId(Long projectId) { + this.projectId = projectId; + } + + @Override + public Long getProjectId() { + return projectId; + } + + @Override + public String getProjectName() { + return projectName; + } + + @Override + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + @Override + public String getOrcName() { + return orcName; + } + + @Override + public void setOrcName(String orcName) { + this.orcName = orcName; + } + + @Override + public Long getOrcId() { + return orcId; + } + + @Override + public void setOrcId(Long orcId) { + this.orcId = orcId; + } + + @Override + public Workspace getWorkspace() { + return workspace; + } + + @Override + public void setWorkspace(Workspace workspace) { + this.workspace = workspace; + } + + @Override + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + @Override + public String getWorkspaceName() { + return this.workspaceName; + } + + @Override + public void setDSSLabels(List dssLabels) { + this.dssLabelList = dssLabels; + } + + @Override + public List getDSSLabels() { + return dssLabelList; + } + + @Override + public String getContextID() { + return contextID; + } + + @Override + public void setContextID(String contextID) { + this.contextID = contextID; + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/impl/NodeRequestRefImpl.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/impl/NodeRequestRefImpl.java new file mode 100644 index 000000000..859421dd6 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/ref/impl/NodeRequestRefImpl.java @@ -0,0 +1,57 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.ref.impl; + +import com.webank.wedatasphere.dss.standard.app.development.ref.NodeRequestRef; + +import java.util.Map; + + +public class NodeRequestRefImpl extends CommonRequestRefImpl implements NodeRequestRef{ + + protected String nodeType; + + protected Map jobContent; + + @Override + public String getNodeType() { + return nodeType; + } + + @Override + public void setNodeType(String nodeType) { + this.nodeType = nodeType; + } + + @Override + public Map getJobContent() { + return jobContent; + } + + @Override + public void setJobContent(Map jobContent) { + this.jobContent = jobContent; + } + + @Override + public String getName() { + if(name == null){ + name = jobContent.get("title").toString(); + } + return name; + } +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractDevelopmentService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractDevelopmentService.java new file mode 100644 index 000000000..f4c577f19 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractDevelopmentService.java @@ -0,0 +1,55 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.app.development.operation.DevelopmentOperation; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppSingletonIntegrationServiceImpl; +import java.util.ArrayList; +import java.util.List; + +public abstract class AbstractDevelopmentService extends AppSingletonIntegrationServiceImpl implements DevelopmentService { + + private List dssLabels = new ArrayList<>(); + private DevelopmentIntegrationStandard appStandard; + + public void setLabels(List labels) { + dssLabels = labels; + } + + @Override + public void setAppStandard(DevelopmentIntegrationStandard appStandard) { + this.appStandard = appStandard; + } + + @Override + public DevelopmentIntegrationStandard getAppStandard() { + return appStandard; + } + + @Override + public List getLabels() { + return dssLabels; + } + + @Override + protected void initOperation(DevelopmentOperation operation) { + operation.setDevelopmentService(this); + } +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefCRUDService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefCRUDService.java new file mode 100644 index 000000000..2732a0d32 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefCRUDService.java @@ -0,0 +1,58 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCopyOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.CopyRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CreateRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.DeleteRequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; + +public abstract class AbstractRefCRUDService extends AbstractDevelopmentService implements RefCRUDService { + + protected abstract RefCreationOperation createRefCreationOperation(); + + protected abstract RefCopyOperation createRefCopyOperation(); + + protected abstract RefUpdateOperation createRefUpdateOperation(); + + protected abstract RefDeletionOperation createRefDeletionOperation(); + + @Override + public RefCreationOperation getRefCreationOperation() { + return getOrCreate(this::createRefCreationOperation, RefCreationOperation.class); + } + + @Override + public RefCopyOperation getRefCopyOperation() { + return getOrCreate(this::createRefCopyOperation, RefCopyOperation.class); + } + + @Override + public RefUpdateOperation getRefUpdateOperation() { + return getOrCreate(this::createRefUpdateOperation, RefUpdateOperation.class); + } + + @Override + public RefDeletionOperation getRefDeletionOperation() { + return getOrCreate(this::createRefDeletionOperation, RefDeletionOperation.class); + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefExecutionService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefExecutionService.java new file mode 100644 index 000000000..a5fb4de90 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefExecutionService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; + +public abstract class AbstractRefExecutionService extends AbstractDevelopmentService implements RefExecutionService { + + public abstract RefExecutionOperation createRefExecutionOperation(); + + @Override + public RefExecutionOperation getRefExecutionOperation() { + return getOrCreate(this::createRefExecutionOperation, RefExecutionOperation.class); + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefExportService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefExportService.java new file mode 100644 index 000000000..0b4c8922e --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefExportService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +public abstract class AbstractRefExportService extends AbstractDevelopmentService implements RefExportService { + + protected abstract RefExportOperation createRefExportOperation(); + + @Override + public RefExportOperation getRefExportOperation() { + return getOrCreate(this::createRefExportOperation, RefExportOperation.class); + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefImportService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefImportService.java new file mode 100644 index 000000000..440e61221 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefImportService.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public abstract class AbstractRefImportService extends AbstractDevelopmentService implements RefImportService { + + protected abstract RefImportOperation createRefImportOperation(); + + @Override + public RefImportOperation getRefImportOperation() { + return getOrCreate(this::createRefImportOperation, RefImportOperation.class); + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefQueryService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefQueryService.java new file mode 100644 index 000000000..78ee098de --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/AbstractRefQueryService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; + + +public abstract class AbstractRefQueryService extends AbstractDevelopmentService implements RefQueryService { + + protected abstract RefQueryOperation createRefQueryOperation(); + + @Override + public RefQueryOperation getRefQueryOperation() { + return getOrCreate(this::createRefQueryOperation, RefQueryOperation.class); + } +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/DevelopmentService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/DevelopmentService.java new file mode 100644 index 000000000..3b3b9e694 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/DevelopmentService.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.app.development.standard.DevelopmentIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppIntegrationService; +import java.util.List; + +public interface DevelopmentService extends AppIntegrationService { + + void setAppStandard(DevelopmentIntegrationStandard appStandard); + + DevelopmentIntegrationStandard getAppStandard(); + + /** + * Labels by default. + * @return All default labels + */ + List getLabels(); +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefCRUDService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefCRUDService.java new file mode 100644 index 000000000..0f7eddead --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefCRUDService.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCopyOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefCreationOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefDeletionOperation; +import com.webank.wedatasphere.dss.standard.app.development.operation.RefUpdateOperation; +import com.webank.wedatasphere.dss.standard.app.development.ref.CopyRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.CreateRequestRef; +import com.webank.wedatasphere.dss.standard.app.development.ref.DeleteRequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +public interface RefCRUDService extends DevelopmentService { + + RefCreationOperation getRefCreationOperation(); + + RefCopyOperation getRefCopyOperation(); + + RefUpdateOperation getRefUpdateOperation(); + + RefDeletionOperation getRefDeletionOperation(); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefExecutionService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefExecutionService.java new file mode 100644 index 000000000..1d460fda0 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefExecutionService.java @@ -0,0 +1,24 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExecutionOperation; + + +public interface RefExecutionService extends DevelopmentService { + RefExecutionOperation getRefExecutionOperation(); +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefExportService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefExportService.java new file mode 100644 index 000000000..86020cd28 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefExportService.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefExportOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + + +public interface RefExportService extends DevelopmentService { + RefExportOperation getRefExportOperation(); +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefImportService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefImportService.java new file mode 100644 index 000000000..dec570bec --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefImportService.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefImportOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +public interface RefImportService extends DevelopmentService { + + RefImportOperation getRefImportOperation(); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefQueryService.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefQueryService.java new file mode 100644 index 000000000..cb8f4302c --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/service/RefQueryService.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.service; + +import com.webank.wedatasphere.dss.standard.app.development.operation.RefQueryOperation; + +public interface RefQueryService extends DevelopmentService { + + RefQueryOperation getRefQueryOperation(); + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/AbstractDevelopmentIntegrationStandard.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/AbstractDevelopmentIntegrationStandard.java new file mode 100644 index 000000000..cc2e94334 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/AbstractDevelopmentIntegrationStandard.java @@ -0,0 +1,80 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.standard; + +import com.webank.wedatasphere.dss.standard.app.development.service.*; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.core.AbstractAppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; + +import java.io.IOException; + +public abstract class AbstractDevelopmentIntegrationStandard + extends AbstractAppIntegrationStandard implements DevelopmentIntegrationStandard { + + @Override + protected void initService(T service) { + service.setAppStandard(this); + } + + @Override + public RefCRUDService getRefCRUDService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRefCRUDService, RefCRUDService.class); + } + + protected abstract RefCRUDService createRefCRUDService(); + + @Override + public RefExecutionService getRefExecutionService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRefExecutionService, RefExecutionService.class); + } + + protected abstract RefExecutionService createRefExecutionService(); + + @Override + public RefExportService getRefExportService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRefExportService, RefExportService.class); + } + + protected abstract RefExportService createRefExportService(); + + @Override + public RefImportService getRefImportService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRefImportService, RefImportService.class); + } + + protected abstract RefImportService createRefImportService(); + + @Override + public RefQueryService getRefQueryService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRefQueryService, RefQueryService.class); + } + + protected abstract RefQueryService createRefQueryService(); + + + @Override + public void init() throws AppStandardErrorException { + + } + + @Override + public void close() throws IOException { + + } +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/DevelopmentIntegrationStandard.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/DevelopmentIntegrationStandard.java new file mode 100644 index 000000000..4bcf941c1 --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/DevelopmentIntegrationStandard.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.standard; + + +import com.webank.wedatasphere.dss.standard.app.development.service.*; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.core.AppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; + +public interface DevelopmentIntegrationStandard extends AppIntegrationStandard { + + RefCRUDService getRefCRUDService(AppInstance appInstance); + + RefExecutionService getRefExecutionService(AppInstance appInstance); + + RefExportService getRefExportService(AppInstance appInstance); + + RefImportService getRefImportService(AppInstance appInstance); + + RefQueryService getRefQueryService(AppInstance appInstance); + + + @Override + default String getStandardName() { + return "developmentIntegrationStandard"; + } + + @Override + default int getGrade() { + return 3; + } + + @Override + default boolean isNecessary() { + return false; + } + +} diff --git a/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/OnlyExecutionDevelopmentStandard.java b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/OnlyExecutionDevelopmentStandard.java new file mode 100644 index 000000000..4bb79270e --- /dev/null +++ b/dss-standard/development-standard/development-process-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/development/standard/OnlyExecutionDevelopmentStandard.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.development.standard; + +import com.webank.wedatasphere.dss.standard.app.development.service.*; + + +public abstract class OnlyExecutionDevelopmentStandard extends AbstractDevelopmentIntegrationStandard { + + @Override + protected RefCRUDService createRefCRUDService() { + return null; +} + + @Override + protected RefExportService createRefExportService() { + return null; + } + + @Override + protected RefImportService createRefImportService() { + return null; + } + + @Override + protected RefQueryService createRefQueryService() { + return null; + } + +} diff --git a/dss-standard/dss-standard-common/pom.xml b/dss-standard/dss-standard-common/pom.xml new file mode 100644 index 000000000..928022db0 --- /dev/null +++ b/dss-standard/dss-standard-common/pom.xml @@ -0,0 +1,104 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../pom.xml + + 4.0.0 + + dss-standard-common + + + + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + provided + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + provided + + + + org.reflections + reflections + ${reflections.version} + + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + **/*.yml + **/*.properties + **/*.sh + **/log4j2.xml + + + + + org.apache.maven.plugins + maven-compiler-plugin + + 8 + 8 + + + + + + src/main/resources + + + ${project.artifactId}-${project.version} + + + + + product + + 1.0.0-SNAPSHOT + + + + + \ No newline at end of file diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppIntegrationService.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppIntegrationService.java new file mode 100644 index 000000000..268a5b237 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppIntegrationService.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.app; + +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.service.AppService; + + +public interface AppIntegrationService extends AppService { + + void setSSORequestService(T ssoRequestService); + + T getSSORequestService(); + + AppInstance getAppInstance(); + + void setAppInstance(AppInstance appInstance); + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppIntegrationServiceImpl.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppIntegrationServiceImpl.java new file mode 100644 index 000000000..04c1dc798 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppIntegrationServiceImpl.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.app; + + +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.service.AppService; +import com.webank.wedatasphere.dss.standard.common.service.AppServiceImpl; + + +public class AppIntegrationServiceImpl extends AppServiceImpl implements AppIntegrationService { + + private T ssoRequestService; + private AppInstance appInstance; + + @Override + public void setSSORequestService(T ssoRequestService) { + this.ssoRequestService = ssoRequestService; + } + + @Override + public T getSSORequestService() { + return ssoRequestService; + } + + @Override + public AppInstance getAppInstance() { + return appInstance; + } + + @Override + public void setAppInstance(AppInstance appInstance) { + this.appInstance = appInstance; + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppSingletonIntegrationServiceImpl.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppSingletonIntegrationServiceImpl.java new file mode 100644 index 000000000..ca831595b --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/app/AppSingletonIntegrationServiceImpl.java @@ -0,0 +1,51 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.app; + +import com.webank.wedatasphere.dss.standard.common.service.AppService; +import com.webank.wedatasphere.dss.standard.common.service.Operation; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.function.Supplier; + + +public class AppSingletonIntegrationServiceImpl extends AppIntegrationServiceImpl { + + private final List appOperations = new ArrayList<>(); + + protected TO getOrCreate(Supplier create, Class clazz) { + Supplier createAndPut = () -> { + TO t = create.get(); + if(t == null) { + return null; + } + initOperation(t); + appOperations.add(t); + return t; + }; + Supplier> filterOperation = () -> appOperations.stream().filter(clazz::isInstance).findFirst().map(operation -> (TO) operation); + return filterOperation.get().orElseGet(() -> { + synchronized (appOperations) { + return filterOperation.get().orElseGet(createAndPut); + } + }); + } + + protected void initOperation(O operation) {} + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AbstractAppIntegrationStandard.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AbstractAppIntegrationStandard.java new file mode 100644 index 000000000..5b23b8016 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AbstractAppIntegrationStandard.java @@ -0,0 +1,71 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.core; + +import com.webank.wedatasphere.dss.standard.common.app.AppIntegrationService; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.service.AppService; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.Supplier; + + +public abstract class AbstractAppIntegrationStandard + implements AppIntegrationStandard { + + private final Map> appServices = new HashMap<>(); + private SSORequestService ssoRequestService; + + protected T getOrCreate(AppInstance appInstance, Supplier create, Class clazz) { + Supplier createAndPut = () -> { + T t = create.get(); + if(t == null) { + return null; + } + t.setSSORequestService(ssoRequestService); + t.setAppInstance(appInstance); + initService(t); + appServices.get(appInstance).add(t); + return t; + }; + if(!appServices.containsKey(appInstance)) { + synchronized (appServices) { + if(!appServices.containsKey(appInstance)) { + appServices.put(appInstance, new ArrayList<>()); + return createAndPut.get(); + } + } + } + final List services = appServices.get(appInstance); + Supplier> filterService = () -> services.stream().filter(clazz::isInstance).findFirst().map(service -> (T) service); + return filterService.get().orElseGet(() -> { + synchronized (services) { + return filterService.get().orElseGet(createAndPut); + } + }); + } + + protected void initService(T service) {} + + @Override + public void setSSORequestService(SSORequestService ssoRequestService) { + this.ssoRequestService = ssoRequestService; + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AppIntegrationStandard.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AppIntegrationStandard.java new file mode 100644 index 000000000..8fbe27bae --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AppIntegrationStandard.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.core; + + +import com.webank.wedatasphere.dss.standard.common.service.AppService; + + +public interface AppIntegrationStandard extends AppStandard { + + void setSSORequestService(SSORequestService ssoService); + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AppStandard.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AppStandard.java new file mode 100644 index 000000000..c19af5297 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/core/AppStandard.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.core; + + +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; + +import java.io.Closeable; + + +public interface AppStandard extends Closeable { + + String getStandardName(); + + int getGrade(); + + void init() throws AppStandardErrorException; + + /** + * 是否必须 + * @return true 表示必须，false表示非必须标准 + */ + default boolean isNecessary() { + return false; + } + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppDesc.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppDesc.java new file mode 100644 index 000000000..b5385064b --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppDesc.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.desc; +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.common.exception.NoSuchAppInstanceException; + +import java.util.List; + + +public interface AppDesc { + + String getAppName(); + + List getAppInstances(); + + /** + * 对于每个AppConn而言，会拥有多个AppInstance，如需要获取相应的AppInstance， + * 通过传入的labels来进行匹配。 + * 返回的List实例，其中第0个是匹配程度最大的实例。 + * */ + List getAppInstancesByLabels(List labels) throws NoSuchAppInstanceException; + + void addAppInstance(AppInstance appInstance); + + void removeAppInstance(AppInstance appInstance); + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppDescImpl.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppDescImpl.java new file mode 100644 index 000000000..2623b202a --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppDescImpl.java @@ -0,0 +1,115 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.desc; + + +import com.webank.wedatasphere.dss.common.label.DSSLabel; +import com.webank.wedatasphere.dss.standard.common.exception.NoSuchAppInstanceException; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.List; + + +public class AppDescImpl implements AppDesc { + + private static final Logger LOG = LoggerFactory.getLogger(AppDescImpl.class); + + private String appName; + private List appInstances = new ArrayList<>(); + + @Override + public String getAppName() { + return appName; + } + + public void setAppName(String appName) { + this.appName = appName; + } + + @Override + public List getAppInstances() { + return appInstances; + } + + @Override + public List getAppInstancesByLabels(List labels) throws NoSuchAppInstanceException{ + // todo~！ + // 1. 通过用户自定义的比较器完成的排序，返回最前一个appInstance + // 2. 返回与labels完全匹配的appInstance + AppInstance targetAppInstance = null; + int similarity = 0; + int maxSimilarity = 0; + for(AppInstance appInstance: appInstances) { + similarity = 0; + List targetLabels = appInstance.getLabels(); + for(DSSLabel label: targetLabels) { + for(DSSLabel userLabel: labels) { + // if user's label equal the target label, then the similarity increase. + if (isEqualLabel(userLabel,label)) { + similarity++; + } + // if current similarity bigger than the maxSimilarity. + if (similarity > maxSimilarity) { + maxSimilarity = similarity; + targetAppInstance = appInstance; + } + } + } + } + // if all the labels is different form the target, then return null. + if(maxSimilarity <= 0) { + LOG.error("{} has no such AppInstance machs the labels: {}", appName, labels); + throw new NoSuchAppInstanceException(60002, "No such AppInstance machs the labels."); + } + return Arrays.asList(targetAppInstance); + } + + //判断两个label是否相等 + public boolean isEqualLabel(DSSLabel userLabel,DSSLabel label){ + boolean flag = false; + + DSSLabel tempUserLabel = (DSSLabel) userLabel; + String userEnv = tempUserLabel.getValue().get(tempUserLabel.getLabelKey()); + + DSSLabel tempLabel = (DSSLabel) label; + String instanceEnv = tempLabel.getValue().get(tempLabel.getLabelKey()); + + if (StringUtils.isNotBlank(userEnv) && userEnv.equalsIgnoreCase(instanceEnv)) { + flag = true; + } + return flag; + } + + public List getAppInstancesByLabels(List labels, Comparator comparator) { + return null; + } + + @Override + public void addAppInstance(AppInstance appInstance) { + appInstances.add(appInstance); + } + + @Override + public void removeAppInstance(AppInstance appInstance) { + appInstances.remove(appInstance); + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppInstance.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppInstance.java new file mode 100644 index 000000000..bf4b5eac2 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppInstance.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.desc; +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; +import java.util.Map; + + +public interface AppInstance { + + Long getId(); + + String getBaseUrl(); + + Map getConfig(); + + List getLabels(); + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppInstanceImpl.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppInstanceImpl.java new file mode 100644 index 000000000..97d37bd75 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/desc/AppInstanceImpl.java @@ -0,0 +1,85 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.desc; + +import com.google.common.base.Objects; +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.List; +import java.util.Map; + + +public class AppInstanceImpl implements AppInstance { + + private Long id; + private String baseUrl; + private Map config; + private List labels; + + @Override + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + @Override + public String getBaseUrl() { + return baseUrl; + } + + public void setBaseUrl(String baseUrl) { + this.baseUrl = baseUrl; + } + + @Override + public Map getConfig() { + return config; + } + + public void setConfig(Map config) { + this.config = config; + } + + @Override + public List getLabels() { + return labels; + } + + public void setLabels(List labels) { + this.labels = labels; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + AppInstanceImpl that = (AppInstanceImpl) o; + return Objects.equal(id, that.id); + } + + @Override + public int hashCode() { + return Objects.hashCode(id); + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AbstractRequestRef.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AbstractRequestRef.java new file mode 100644 index 000000000..ac9f95f43 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AbstractRequestRef.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import java.util.HashMap; +import java.util.Map; + + +public abstract class AbstractRequestRef implements RequestRef { + + private Map parameters = new HashMap<>(); + protected String name; + protected String type; + + @Override + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Override + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + @Override + public Object getParameter(String key) { + return parameters.get(key); + } + + @Override + public void setParameter(String key, Object value) { + parameters.put(key, value); + } + + @Override + public Map getParameters() { + return parameters; + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AbstractResponseRef.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AbstractResponseRef.java new file mode 100644 index 000000000..c140caf34 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AbstractResponseRef.java @@ -0,0 +1,53 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import java.util.Map; + + +public abstract class AbstractResponseRef implements ResponseRef { + + protected String responseBody; + protected int status = -1; + protected String errorMsg; + protected Map responseMap; + + protected AbstractResponseRef(String responseBody, int status) { + this.responseBody = responseBody; + this.status = status; + } + + @Override + public Object getValue(String key) { + return toMap().get(key); + } + + @Override + public String getResponseBody() { + return responseBody; + } + + @Override + public int getStatus() { + return status; + } + + @Override + public boolean isSucceed() { + return status == 0 || status == 200; + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AppConnRefFactoryUtils.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AppConnRefFactoryUtils.java new file mode 100644 index 000000000..ec158e849 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AppConnRefFactoryUtils.java @@ -0,0 +1,39 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; + + +public class AppConnRefFactoryUtils { + + public static final String APP_CONN_PACKAGE_HEADER = CommonVars.apply("wds.dss.appconn.package.header", "com.webank.wedatasphere.dss.appconn.").getValue(); + + public static R newAppConnRef(Class clazz, String appConnName) throws DSSErrorException { + return RefFactory.INSTANCE.newRef(clazz, Thread.currentThread().getContextClassLoader(), APP_CONN_PACKAGE_HEADER + appConnName); + } + + public static R newAppConnRef(Class clazz, ClassLoader classLoader, String appConnName) throws DSSErrorException { + return RefFactory.INSTANCE.newRef(clazz, classLoader, APP_CONN_PACKAGE_HEADER + appConnName); + } + + public static R newAppConnRefByPackageName(Class clazz, ClassLoader classLoader, String packageName) throws DSSErrorException { + return RefFactory.INSTANCE.newRef(clazz, classLoader, packageName); + } + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AsyncResponseRef.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AsyncResponseRef.java new file mode 100644 index 000000000..4a876923e --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/AsyncResponseRef.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import java.util.function.Consumer; + + +public interface AsyncResponseRef extends ResponseRef { + + long getStartTime(); + + boolean isCompleted(); + + ResponseRef getResponse(); + + void waitForCompleted() throws InterruptedException; + + void notifyMe(Consumer notifyListener); + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/CommonResponseRef.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/CommonResponseRef.java new file mode 100644 index 000000000..16a0a5931 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/CommonResponseRef.java @@ -0,0 +1,83 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import com.webank.wedatasphere.dss.common.utils.DSSCommonUtils; +import java.util.HashMap; +import java.util.Map; +import org.apache.commons.lang.StringUtils; + + +public class CommonResponseRef extends AbstractResponseRef { + + public CommonResponseRef(String responseBody, int status) { + super(responseBody, status); + init(); + } + + protected void init() { + if(StringUtils.isNotBlank(responseBody)) { + responseMap = DSSCommonUtils.COMMON_GSON.fromJson(responseBody, Map.class); + } else { + responseMap = new HashMap<>(); + } + } + + public CommonResponseRef(){ + this("",0); + } + + @Override + public Map toMap() { + return responseMap; + } + + @Override + public String getErrorMsg() { + return errorMsg; + } + + @Override + public Object getValue(String key) { + return this.responseMap.get(key); + } + + public void addResponse(String key, Object value){ + this.responseMap.put(key, value); + } + + public static CommonResponseRef success() { + return new CommonResponseRef(); + } + + public static CommonResponseRef success(String message) { + CommonResponseRef commonResponseRef = new CommonResponseRef(); + commonResponseRef.responseBody = message; + return commonResponseRef; + } + + public static CommonResponseRef error(String errorMsg, Throwable cause) { + CommonResponseRef commonResponseRef = new CommonResponseRef(); + commonResponseRef.status = 1; + commonResponseRef.errorMsg = errorMsg; + return commonResponseRef; + } + + public static CommonResponseRef error(String errorMsg) { + return error(errorMsg, null); + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/DefaultRefFactory.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/DefaultRefFactory.java new file mode 100644 index 000000000..2759cd727 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/DefaultRefFactory.java @@ -0,0 +1,131 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; +import com.webank.wedatasphere.dss.common.utils.DSSExceptionUtils; +import com.webank.wedatasphere.linkis.common.utils.ClassUtils; +import java.net.URL; +import java.util.Collection; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.stream.Collectors; +import org.reflections.Reflections; +import org.reflections.scanners.SubTypesScanner; +import org.reflections.util.ClasspathHelper; +import org.reflections.util.ConfigurationBuilder; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class DefaultRefFactory implements RefFactory { + + private static final Logger LOGGER = LoggerFactory.getLogger(DefaultRefFactory.class); + + private class RefKey{ + private ClassLoader classLoader; + private Class clazz; + private String packageName; + + public RefKey(ClassLoader classLoader, Class clazz, String packageName) { + this.classLoader = classLoader; + this.clazz = clazz; + this.packageName = packageName; + } + + @SuppressWarnings("unchecked") + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + RefKey refKey = (RefKey) o; + return com.google.common.base.Objects.equal(classLoader, refKey.classLoader) && + com.google.common.base.Objects.equal(clazz, refKey.clazz) && + com.google.common.base.Objects.equal(packageName, refKey.packageName); + } + + @Override + public int hashCode() { + return com.google.common.base.Objects.hashCode(classLoader, clazz, packageName); + } + } + + + + private Map> cacheMap = new ConcurrentHashMap<>(16); + + + @Override + public R newRef(Class clazz) throws DSSErrorException { + if(!ClassUtils.isInterfaceOrAbstract(clazz)) { + return DSSExceptionUtils.tryAndWarn(Void -> clazz.newInstance()); + } else { + return com.webank.wedatasphere.dss.common.utils.ClassUtils.getInstance(clazz); + } + } + + @Override + public R newRef(Class clazz, ClassLoader classLoader, String packageName) throws DSSErrorException { + RefKey refKey = new RefKey(classLoader, clazz, packageName); + Class refClass = cacheMap.get(refKey); + if(cacheMap.containsKey(refKey) && refClass != null){ + return DSSExceptionUtils.tryAndWarn(Void -> (R) refClass.newInstance()); + } else if(cacheMap.containsKey(refKey) && refClass == null) { + return newRef(clazz); + } + ConfigurationBuilder configurationBuilder = new ConfigurationBuilder(); + configurationBuilder.addClassLoader(classLoader); + configurationBuilder.addClassLoader(clazz.getClassLoader()); + configurationBuilder.forPackages(packageName); + Collection urls = ClasspathHelper.forClassLoader(classLoader); + configurationBuilder.addUrls(urls.stream().filter(Objects::nonNull).collect(Collectors.toList())); + configurationBuilder.addScanners(new SubTypesScanner()); + Reflections reflections = new Reflections(configurationBuilder); + Set> subClasses = reflections.getSubTypesOf(clazz).stream() + .filter(c -> !ClassUtils.isInterfaceOrAbstract(c)).collect(Collectors.toSet()); + if(subClasses.isEmpty()){ + // Put null, so just search it in one time. + cacheMap.put(refKey, null); + return newRef(clazz); + } else { + Set> realSubClasses = subClasses; + if (subClasses.size() > 1) { + LOGGER.warn("subClass of {} size is {}, classes are {}", clazz.getName(), subClasses.size(), subClasses); + realSubClasses = subClasses.stream().filter(subClass -> subClass.getName().contains(packageName) && !((Class) subClass).isInterface()).collect(Collectors.toSet()); + LOGGER.warn("realSubClasses is {} ", realSubClasses); + } + if (realSubClasses.size() > 1){ + LOGGER.error("realSubClasses size is bigger than 1"); + DSSExceptionUtils.dealErrorException(60091, "too many subclass of " + clazz.getName() + "in " + packageName, + DSSErrorException.class); + } else if(realSubClasses.size() == 0) { + // Put null, so just search it in one time. + cacheMap.put(refKey, null); + return newRef(clazz); + } + Class subClass = realSubClasses.iterator().next(); + cacheMap.put(refKey, subClass); + return DSSExceptionUtils.tryAndWarn(Void -> subClass.newInstance()); + } + } + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/Ref.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/Ref.java new file mode 100644 index 000000000..6be5749e9 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/Ref.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +public interface Ref { + + @Override + boolean equals(Object ref); + + @Override + String toString(); + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/RefFactory.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/RefFactory.java new file mode 100644 index 000000000..1c24d1c8e --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/RefFactory.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + + +public interface RefFactory { + + RefFactory INSTANCE = new DefaultRefFactory(); + + R newRef(Class clazz) throws DSSErrorException; + + /** + * We need to instantiate Ref, because each Ref is instantiated in a different AppConn, + * so their ClassLoader is also different from the main ClassLoader, + * so we need to pass in the ClassLoader that instantiates the AppConn. + * @param clazz The interface inherited by the class that needs to be instantiated. + * @param classLoader Instantiate the classloader of appConn. + * @param packageName package name. + * @return return a instance of R. + * @throws DSSErrorException + */ + R newRef(Class clazz, ClassLoader classLoader, String packageName) throws DSSErrorException; + +} \ No newline at end of file diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/RequestRef.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/RequestRef.java new file mode 100644 index 000000000..c9ba5c8bf --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/RequestRef.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; +import com.webank.wedatasphere.dss.common.label.DSSLabel; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + + +public interface RequestRef extends Ref { + + Object getParameter(String key); + + void setParameter(String key, Object value); + + Map getParameters(); + + String getName(); + + String getType(); + + default List getDSSLabels(){ + return new ArrayList<>(); + } + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/ResponseRef.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/ResponseRef.java new file mode 100644 index 000000000..17fba78ce --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/entity/ref/ResponseRef.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.entity.ref; + +import java.util.Map; + + +public interface ResponseRef extends Ref { + + Object getValue(String key); + + Map toMap(); + + String getResponseBody(); + + int getStatus(); + + String getErrorMsg(); + + boolean isSucceed(); + + default boolean isFailed() { + return !isSucceed(); + } + +} \ No newline at end of file diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/AppStandardErrorException.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/AppStandardErrorException.java new file mode 100644 index 000000000..f8ddbffe6 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/AppStandardErrorException.java @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.exception; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + + +public class AppStandardErrorException extends DSSErrorException { + + + public AppStandardErrorException(int errorCode, String errorMessage){ + super(errorCode,errorMessage); + } + + public AppStandardErrorException(int errorCode, String message, Throwable cause) { + super(errorCode, message); + this.initCause(cause); + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/AppStandardWarnException.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/AppStandardWarnException.java new file mode 100644 index 000000000..40dc940af --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/AppStandardWarnException.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.exception; + + +public class AppStandardWarnException extends RuntimeException { + + public AppStandardWarnException(int errorCode, String message) { + super(message); + } + + public AppStandardWarnException(int errorCode, String message, Throwable cause) { + super(message, cause); + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/NoSuchAppInstanceException.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/NoSuchAppInstanceException.java new file mode 100644 index 000000000..5e1668dbf --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/NoSuchAppInstanceException.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.exception; + +import com.webank.wedatasphere.dss.common.exception.DSSErrorException; + + +public class NoSuchAppInstanceException extends DSSErrorException { + + public NoSuchAppInstanceException(int errCode, String desc) { + super(errCode, desc); + } + + public NoSuchAppInstanceException(int errCode, String desc, String ip, int port, String serviceKind) { + super(errCode, desc, ip, port, serviceKind); + } + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/operation/ExternalOperationFailedException.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/operation/ExternalOperationFailedException.java new file mode 100644 index 000000000..940d92ce7 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/exception/operation/ExternalOperationFailedException.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.exception.operation; + +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; + +public class ExternalOperationFailedException extends AppStandardErrorException { + + public ExternalOperationFailedException(int errorCode, String message){ + super(errorCode, message); + } + + public ExternalOperationFailedException(int errorCode, String message, Throwable cause) { + super(errorCode, message, cause); + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/AppService.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/AppService.java new file mode 100644 index 000000000..330fc01ac --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/AppService.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.service; + + +public interface AppService { + + Operation createOperation(Class clazz); + + boolean isOperationExists(Class clazz); + + boolean isOperationNecessary(Class clazz); + +} \ No newline at end of file diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/AppServiceImpl.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/AppServiceImpl.java new file mode 100644 index 000000000..daaa64f45 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/AppServiceImpl.java @@ -0,0 +1,82 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.service; + +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardWarnException; + +import java.lang.reflect.Method; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; + + +public class AppServiceImpl implements AppService { + + private Set> necessaryOperations = new HashSet<>(); + + protected void registerNecessaryOperation(Class clazz) { + necessaryOperations.add(clazz); + } + + @Override + public Operation createOperation(Class clazz) { + String clazzSimpleName = clazz.getSimpleName(); + List methodList = Arrays.stream(this.getClass().getDeclaredMethods()) + .filter(method -> method.getReturnType() == clazz + && method.getParameterCount() == 0).collect(Collectors.toList()); + if(methodList.size() == 1) { + try { + return (Operation) methodList.get(0).invoke(this); + } catch (ReflectiveOperationException e) { + throw new AppStandardWarnException(80020, "Not exists operation: " + clazzSimpleName, e); + } + } else if(methodList.isEmpty()) { + return notFoundOperation(clazz); + } else { + return multiFoundOperation(clazz); + } + } + + protected Operation notFoundOperation(Class clazz) { + throw new AppStandardWarnException(80020, "Not exists operation: " + clazz.getSimpleName()); + } + + protected Operation multiFoundOperation(Class clazz) { + throw new AppStandardWarnException(80020, "Multi exists operations: " + clazz.getSimpleName()); + } + + @Override + public boolean isOperationExists(Class clazz) { + try{ + return createOperation(clazz) != null; + } catch (AppStandardWarnException e) { + return false; + } + } + + @Override + public boolean isOperationNecessary(Class clazz) { + boolean isNecessary = necessaryOperations.contains(clazz); + if(isNecessary) { + return true; + } else { + return isOperationExists(clazz); + } + } +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/Operation.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/Operation.java new file mode 100644 index 000000000..236e88787 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/service/Operation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.service; + + +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +/** + * Operation是顶层的操作类,对于操作来说，传入一个RequestRef,返回一个ResponseRef + */ +public interface Operation { + +} diff --git a/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/utils/AppStandardClassUtils.java b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/utils/AppStandardClassUtils.java new file mode 100644 index 000000000..5778e5b28 --- /dev/null +++ b/dss-standard/dss-standard-common/src/main/java/com/webank/wedatasphere/dss/standard/common/utils/AppStandardClassUtils.java @@ -0,0 +1,81 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.common.utils; + +import com.webank.wedatasphere.dss.common.utils.ClassUtils.ClassHelper; +import java.util.HashMap; +import java.util.Map; +import java.util.function.Supplier; +import org.reflections.Reflections; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * This class is defined for AppConn jar, if some classes in AppConn want to load a class in AppConn jar. + */ +public class AppStandardClassUtils extends ClassHelper { + + private static final Map INSTANCES = new HashMap<>(); + private static final Map CLASS_LOADER_MAP = new HashMap<>(); + + private static final Logger LOGGER = LoggerFactory.getLogger(AppStandardClassUtils.class); + + public static ClassLoader getClassLoader(String appConnName, Supplier createClassLoader) { + if(!CLASS_LOADER_MAP.containsKey(appConnName)) { + synchronized (AppStandardClassUtils.class) { + if(!CLASS_LOADER_MAP.containsKey(appConnName)) { + CLASS_LOADER_MAP.put(appConnName, createClassLoader.get()); + LOGGER.info("Has stored {} ClassLoader.", CLASS_LOADER_MAP.size()); + } + } + } + return CLASS_LOADER_MAP.get(appConnName); + } + + public static AppStandardClassUtils getInstance(String appConnName) { + if(!INSTANCES.containsKey(appConnName)) { + synchronized (AppStandardClassUtils.class) { + if(!INSTANCES.containsKey(appConnName)) { + INSTANCES.put(appConnName, new AppStandardClassUtils(appConnName)); + LOGGER.info("Has stored {} AppStandardClassUtils.", INSTANCES.size()); + } + } + } + return INSTANCES.get(appConnName); + } + + private Reflections reflection; + private String appConnName; + + private AppStandardClassUtils(String appConnName) { + this.appConnName = appConnName; + } + + @Override + protected Reflections getReflections(Class clazz) { + if(reflection == null) { + synchronized (this) { + if(reflection == null) { + ClassLoader classLoader = clazz.getClassLoader(); + reflection = new Reflections("com.webank.wedatasphere.dss", CLASS_LOADER_MAP.get(appConnName), classLoader); + } + } + } + return reflection; + } + +} diff --git a/dss-standard/pom.xml b/dss-standard/pom.xml new file mode 100644 index 000000000..0c46bebec --- /dev/null +++ b/dss-standard/pom.xml @@ -0,0 +1,45 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../pom.xml + + 4.0.0 + + dss-standard + pom + + + dss-standard-common + sso-standard/sso-integration-standard + sso-standard/origin-sso-integration-standard + sso-standard/spring-origin-sso-integration-plugin + structure-standard/dss-structure-integration-standard + structure-standard/dss-project-plugin + structure-standard/spring-origin-dss-project-plugin + structure-standard/dss-role-plugin + development-standard/development-process-standard + development-standard/development-process-standard-execution + + + \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/pom.xml b/dss-standard/sso-standard/origin-sso-integration-standard/pom.xml new file mode 100644 index 000000000..da49b5497 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/pom.xml @@ -0,0 +1,76 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-origin-sso-integration-standard + + + + com.webank.wedatasphere.linkis + linkis-gateway-httpclient-support + ${linkis.version} + + + javax.servlet + javax.servlet-api + 3.1.0 + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + + + \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/OriginSSOIntegrationStandard.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/OriginSSOIntegrationStandard.scala new file mode 100644 index 000000000..31039d112 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/OriginSSOIntegrationStandard.scala @@ -0,0 +1,55 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin + + +import com.webank.wedatasphere.dss.standard.app.sso.{SSOIntegrationStandard, SSOIntegrationStandardFactory} +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient +import com.webank.wedatasphere.dss.standard.app.sso.origin.plugin.OriginSSOPluginServiceImpl +import com.webank.wedatasphere.dss.standard.app.sso.origin.request.OriginSSORequestServiceImpl +import com.webank.wedatasphere.dss.standard.app.sso.plugin.SSOPluginService +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService +import com.webank.wedatasphere.dss.standard.common.desc.AppDesc + + +class OriginSSOIntegrationStandard private[origin]() extends SSOIntegrationStandard { + + private val ssoPluginService: SSOPluginService = new OriginSSOPluginServiceImpl + private val ssoRequestService: SSORequestService = new OriginSSORequestServiceImpl + + override def getSSORequestService: SSORequestService = ssoRequestService + + override def getSSOPluginService: SSOPluginService = ssoPluginService + + override def init(): Unit = { + ssoPluginService.setSSOBuilderService(getSSOBuilderService) + } + + override def close(): Unit = HttpClient.close() +} + +// SSO 工厂，通过该工程获取sso集成规范 +class OriginSSOIntegrationStandardFactory extends SSOIntegrationStandardFactory { + + private val ssoIntegrationStandard = new OriginSSOIntegrationStandard + + override def init(): Unit = { + ssoIntegrationStandard.init() + } + + override def getSSOIntegrationStandard: SSOIntegrationStandard = ssoIntegrationStandard +} \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/client/HttpClient.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/client/HttpClient.scala new file mode 100644 index 000000000..a14c9a6bf --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/client/HttpClient.scala @@ -0,0 +1,114 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.client + +import java.net.URI +import java.text.SimpleDateFormat +import java.util +import java.util.Date + +import com.fasterxml.jackson.databind.ObjectMapper +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import com.webank.wedatasphere.linkis.httpclient.Client +import com.webank.wedatasphere.linkis.httpclient.dws.DWSHttpClient +import com.webank.wedatasphere.linkis.httpclient.dws.config.DWSClientConfigBuilder +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction +import org.apache.commons.io.IOUtils +import org.apache.http.impl.cookie.BasicClientCookie + +import scala.collection.JavaConversions._ + + +object HttpClient extends Logging { + + private val dssClients = new util.HashMap[String, DWSHttpClient] + private val httpClients = new util.HashMap[String, Client] + + + + def getBaseUrl(url: String): String = { + val uri = new URI(url) + if (uri.getPort > 0){ + uri.getScheme + "://" + uri.getHost + ":" + uri.getPort + } else { + uri.getScheme + "://" + uri.getHost + } + } + + private def getClient[T](url: String, cacheMap: util.HashMap[String, T], createNewClient: String => T): T = { + val baseUrl = getBaseUrl(url) + if(!cacheMap.containsKey(baseUrl)) baseUrl.intern synchronized { + if(!cacheMap.containsKey(baseUrl)) { + info("create a new Client for url " + baseUrl) + val client = createNewClient(baseUrl) + cacheMap.put(baseUrl, client) + } + } + cacheMap.get(baseUrl) + } + + def getHttpClient(url: String, appName: String): Client = { + val baseUrl = getBaseUrl(url) + val clientConfig = DWSClientConfigBuilder. + newBuilder(). + addServerUrl(baseUrl). + connectionTimeout(connectTimeout). + setDWSVersion("v1"). + discoveryEnabled(false). + maxConnectionSize(maxConnection). + readTimeout(readTimeout).build() + new DWSHttpClient(clientConfig, appName + "-SSO-Client") + } + + def getDSSClient(dssUrl: String): DWSHttpClient = getClient(dssUrl, dssClients, baseUrl => { + val clientConfig = DWSClientConfigBuilder.newBuilder().setDWSVersion(dssVersion) + .addServerUrl(baseUrl).connectionTimeout(connectTimeout).discoveryEnabled(true) + .maxConnectionSize(maxConnection).readTimeout(readTimeout).build() + new DWSHttpClient(clientConfig, "DSS-Integration-Standard-Client") + }) + + def addCookies(dssMsg: DSSMsg, action: HttpAction): Unit = + dssMsg.getCookies.foreach { case (key, value) => + val basicClientCookie = new BasicClientCookie(key, value) + val domain = Utils.tryCatch(new URI(dssMsg.getDSSUrl).getHost)(_ => HttpClient.getBaseUrl(dssMsg.getDSSUrl)) + basicClientCookie.setDomain(domain) + basicClientCookie.setPath("/") + basicClientCookie.setExpiryDate(new Date(System.currentTimeMillis() + 1000 * 60 * 60 * 24 * 30L)) + info("Add cookie for get user info "+basicClientCookie.toString) + action.addCookie(basicClientCookie) + } + + def close(): Unit = { + dssClients.values().foreach(IOUtils.closeQuietly) + httpClients.values().foreach(IOUtils.closeQuietly) + } + + private var dssVersion = "v1" + private var maxConnection = 50 + private var connectTimeout = 300000 + private var readTimeout = 300000 + + //TODO 切换为linkis-common的JsonUtils + val objectMapper = new ObjectMapper().setDateFormat(new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ")) + + def setDSSVersion(version: String): Unit = this.dssVersion = version + def setMaxConnection(maxConnection: Int): Unit = this.maxConnection = maxConnection + def setConnectTimeout(connectTimeout: Int): Unit = this.connectTimeout = connectTimeout + def setReadTimeout(readTimeout: Int): Unit = this.readTimeout = readTimeout + +} diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOMsgParseOperation.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOMsgParseOperation.scala new file mode 100644 index 000000000..444fa6e88 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOMsgParseOperation.scala @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.plugin + +import com.webank.wedatasphere.dss.common.utils.IoUtils +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient +import com.webank.wedatasphere.dss.standard.app.sso.plugin.AbstractSSOMsgParseOperation +import com.webank.wedatasphere.linkis.common.utils.Utils +import com.webank.wedatasphere.linkis.httpclient.Client +import org.apache.commons.io.IOUtils + + +class OriginSSOMsgParseOperation extends AbstractSSOMsgParseOperation { + + override protected def getUser(dssMsg: DSSMsg): String = { + val dssUrl = dssMsg.getDSSUrl + val dwsHttpClient:Client=null + Utils.tryFinally({ + val dwsHttpClient = HttpClient.getHttpClient(dssUrl, "DSS") + val userInfoAction = new UserInfoAction + HttpClient.addCookies(dssMsg, userInfoAction) + dwsHttpClient.execute(userInfoAction) match { + case userInfoResult: UserInfoResult => + userInfoResult.getUserName + } + })(IOUtils.closeQuietly(dwsHttpClient)) + } + +} diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOPluginFilter.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOPluginFilter.scala new file mode 100644 index 000000000..e887e913c --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOPluginFilter.scala @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.plugin + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard +import com.webank.wedatasphere.dss.standard.app.sso.origin.OriginSSOIntegrationStandardFactory +import com.webank.wedatasphere.dss.standard.app.sso.plugin.filter.SSOPluginFilter +import com.webank.wedatasphere.linkis.common.utils.Logging + + +abstract class OriginSSOPluginFilter extends SSOPluginFilter with Logging{ + + private val factory = new OriginSSOIntegrationStandardFactory + + override def info(str: String): Unit = logger.info(str) + + override protected def getSSOIntegrationStandard: SSOIntegrationStandard = factory.getSSOIntegrationStandard +} diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOPluginServiceImpl.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOPluginServiceImpl.scala new file mode 100644 index 000000000..212314226 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginSSOPluginServiceImpl.scala @@ -0,0 +1,43 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.plugin + +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOBuilderService +import com.webank.wedatasphere.dss.standard.app.sso.plugin.{SSOMsgParseOperation, SSOPluginService, WorkspacePlugin} +import com.webank.wedatasphere.dss.standard.common.service.AppServiceImpl +import com.webank.wedatasphere.linkis.common.utils.Logging + + +class OriginSSOPluginServiceImpl extends AppServiceImpl with SSOPluginService with Logging { + + private var ssoBuilderService: SSOBuilderService = _ + private val ssoMsgParseOperation = new OriginSSOMsgParseOperation + private val workspacePlugin = new OriginWorkspacePlugin + workspacePlugin.setDssMsgCacheOperation(createDssMsgCacheOperation()) + + override def setSSOBuilderService(ssoBuilderService: SSOBuilderService): Unit = { + this.ssoBuilderService = ssoBuilderService + ssoMsgParseOperation.setSSOBuilderService(ssoBuilderService) + + } + + override def createSSOMsgParseOperation(): SSOMsgParseOperation = ssoMsgParseOperation + + override def close(): Unit = {} + + override def createWorkspacePluginOperation(): WorkspacePlugin = workspacePlugin +} \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginWorkspacePlugin.java b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginWorkspacePlugin.java new file mode 100644 index 000000000..0e2838654 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/OriginWorkspacePlugin.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.plugin; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.AbstractWorkspacePlugin; +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient; +import com.webank.wedatasphere.linkis.httpclient.dws.DWSHttpClient; +import java.util.List; + + +public class OriginWorkspacePlugin extends AbstractWorkspacePlugin { + + @Override + protected List getAllUsers(DssMsgBuilderOperation.DSSMsg dssMsg) { + String dssUrl = dssMsg.getDSSUrl(); + DWSHttpClient dwsHttpClient = HttpClient.getDSSClient(dssUrl); + WorkspaceUsersAction workspaceUsersAction = new WorkspaceUsersAction(); + HttpClient.addCookies(dssMsg, workspaceUsersAction); + WorkspaceUsersResult result = (WorkspaceUsersResult) dwsHttpClient.execute(workspaceUsersAction); + return result.getUsers(); + } +} diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/UserInfoAction.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/UserInfoAction.scala new file mode 100644 index 000000000..8acf94e10 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/UserInfoAction.scala @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.plugin + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.GetAction +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction + + +class UserInfoAction extends GetAction with DWSHttpAction { + override def suffixURLs: Array[String] = Array("user", "userInfo") +} +class WorkspaceUsersAction extends POSTAction with DWSHttpAction { + override def getRequestPayload: String = "" + + override def suffixURLs: Array[String] = Array("dss", "getUsersOfWorkspace") + + def setWorkspace(workspace: String): Unit = addRequestPayload("workspaceName", workspace) +} \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/UserInfoResult.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/UserInfoResult.scala new file mode 100644 index 000000000..e59ab4ba3 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/plugin/UserInfoResult.scala @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.plugin + +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +import scala.beans.BeanProperty + + +@DWSHttpMessageResult("/api/rest_j/v\\d+/user/userInfo") +class UserInfoResult extends DWSResult { + + @BeanProperty var userName: String = _ + +} + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/getUsersOfWorkspace") +class WorkspaceUsersResult extends DWSResult { + + @BeanProperty var users: java.util.List[String] = _ + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/request/OriginSSORequestOperation.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/request/OriginSSORequestOperation.scala new file mode 100644 index 000000000..e03844121 --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/request/OriginSSORequestOperation.scala @@ -0,0 +1,127 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.request + +import java.net.{URI, URL, URLDecoder} +import java.util +import java.util.Date +import java.util.concurrent.{ConcurrentHashMap, TimeUnit} + +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation +import com.webank.wedatasphere.dss.standard.app.sso.builder.impl.SSOUrlBuilderOperationImpl +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestOperation +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException +import com.webank.wedatasphere.linkis.common.utils.{ByteTimeUtils, Logging, Utils} +import com.webank.wedatasphere.linkis.httpclient.Client +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction +import com.webank.wedatasphere.linkis.httpclient.response.impl.DefaultHttpResult +import com.webank.wedatasphere.linkis.httpclient.response.{HttpResult, Result} +import org.apache.commons.io.IOUtils +import org.apache.commons.lang.StringUtils +import org.apache.http.impl.cookie.BasicClientCookie + +import scala.collection.convert.wrapAsScala._ + + +class OriginSSORequestOperation private[request](appName: String) extends SSORequestOperation[HttpAction, HttpResult] with Logging { + + override def requestWithSSO(urlBuilder: SSOUrlBuilderOperation, req: HttpAction): HttpResult = { + + val httpClient = HttpClient.getHttpClient(urlBuilder.getBuiltUrl, appName) + urlBuilder match { + case urlBuilderOperationImpl: SSOUrlBuilderOperationImpl => val cookies = urlBuilderOperationImpl.getCookies + cookies.foreach { + case (key, value) => + val basicClientCookie = new BasicClientCookie(key, value) + val domain = Utils.tryCatch(new URI(urlBuilder.getBuiltUrl).getHost)(_ => "") + basicClientCookie.setDomain(domain) + basicClientCookie.setPath("/") + basicClientCookie.setExpiryDate(new Date(System.currentTimeMillis() + 1000 * 60 * 60 * 24 * 30L)) + info("Add cookie for get user info " + basicClientCookie.toString) + req.addCookie(basicClientCookie) + } + } + Utils.tryFinally({ + httpClient.execute(req) match { + case result: HttpResult => result + case result => if (Result.isSuccessResult(result)) { + val defaultHttpResult = new DefaultHttpResult + defaultHttpResult.set(null, 200, null, null) + defaultHttpResult + } else throw new AppStandardErrorException(20300, s"Not support Result => ${result.getClass.getName}.") + } + })(IOUtils.closeQuietly(httpClient)) + } + +} + +object OriginSSORequestOperation extends Logging { + + val MAX_ACTIVE_TIME = ByteTimeUtils.timeStringAsMs("15m") + + private val httpClientLastAccessMap = new ConcurrentHashMap[String, Long] + private val httpClientMap = new util.HashMap[String, Client] + + Utils.defaultScheduler.scheduleWithFixedDelay(new Runnable { + override def run(): Unit = httpClientLastAccessMap.keySet().toArray.foreach { + case key: String => + if (System.currentTimeMillis - httpClientLastAccessMap.get(key) >= MAX_ACTIVE_TIME) httpClientMap synchronized { + if (httpClientLastAccessMap.containsKey(key)) { + httpClientLastAccessMap.remove(key) + IOUtils.closeQuietly(httpClientMap.get(key)) + httpClientMap.remove(key) + info(s"SSORequestOperation removed expired key($key).") + } + } + } + }, MAX_ACTIVE_TIME, MAX_ACTIVE_TIME, TimeUnit.MILLISECONDS) + + def getHttpClient(urlBuilder: SSOUrlBuilderOperation, appName: String): Client = urlBuilder match { + case builder: SSOUrlBuilderOperationImpl => + builder.getCookies.find(_._1 == "bdp-user-ticket-id").foreach { case (_, ticketId) => + val key = getKey(ticketId, appName) + if (httpClientMap.containsKey(key) && System.currentTimeMillis - httpClientLastAccessMap.get(key) < MAX_ACTIVE_TIME) { + httpClientLastAccessMap.put(key, System.currentTimeMillis) + return httpClientMap.get(key) + } + else httpClientMap synchronized { + if (httpClientMap.containsKey(key) && System.currentTimeMillis - httpClientLastAccessMap.get(key) < MAX_ACTIVE_TIME) { + httpClientLastAccessMap.put(key, System.currentTimeMillis) + return httpClientMap.get(key) + } else { + if (httpClientMap.containsKey(key)) { + httpClientMap.get(key).close() + } + + val httpClient = HttpClient.getHttpClient(urlBuilder.getBuiltUrl, appName) + httpClientMap.put(key, httpClient) + httpClientLastAccessMap.put(key, System.currentTimeMillis) + info(s"SSORequestOperation add a new HttpClient for key($key).") + return httpClient + } + } + } + throw new AppStandardErrorException(20300, "User has not login, please login first.") + case _ => + throw new AppStandardErrorException(20300, s"Not support SSOUrlBuilderOperation => ${urlBuilder.getClass.getName}.") + } + + + private def getKey(ticketId: String, appName: String): String = appName + ticketId + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/request/OriginSSORequestServiceImpl.scala b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/request/OriginSSORequestServiceImpl.scala new file mode 100644 index 000000000..225322d2e --- /dev/null +++ b/dss-standard/sso-standard/origin-sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/origin/request/OriginSSORequestServiceImpl.scala @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.request + +import java.util + +import com.webank.wedatasphere.dss.standard.app.sso.request.{SSORequestOperation, SSORequestService} +import com.webank.wedatasphere.dss.standard.common.service.AppServiceImpl +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction +import com.webank.wedatasphere.linkis.httpclient.response.HttpResult + + +class OriginSSORequestServiceImpl extends AppServiceImpl with SSORequestService { + + private val ssoRequestServices = new util.HashMap[String, OriginSSORequestOperation] + + override def createSSORequestOperation(appName: String): SSORequestOperation[HttpAction, HttpResult] = { + if(!ssoRequestServices.containsKey(appName)) synchronized { + if(!ssoRequestServices.containsKey(appName)) ssoRequestServices.put(appName, new OriginSSORequestOperation(appName)) + } + ssoRequestServices.get(appName) + } +} diff --git a/dss-standard/sso-standard/spring-origin-sso-integration-plugin/pom.xml b/dss-standard/sso-standard/spring-origin-sso-integration-plugin/pom.xml new file mode 100644 index 000000000..77209785e --- /dev/null +++ b/dss-standard/sso-standard/spring-origin-sso-integration-plugin/pom.xml @@ -0,0 +1,89 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + spring-origin-sso-integration-plugin + + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + com.webank.wedatasphere.dss + dss-origin-sso-integration-standard + ${dss.version} + + + org.springframework + spring-core + ${spring.version} + compile + + + org.springframework + spring-context + ${spring.version} + compile + + + org.springframework + spring-web + ${spring.version} + compile + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + + + \ No newline at end of file diff --git a/dss-standard/sso-standard/spring-origin-sso-integration-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/origin/filter/spring/SpringOriginSSOPluginFilter.java b/dss-standard/sso-standard/spring-origin-sso-integration-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/origin/filter/spring/SpringOriginSSOPluginFilter.java new file mode 100644 index 000000000..3c69165a5 --- /dev/null +++ b/dss-standard/sso-standard/spring-origin-sso-integration-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/origin/filter/spring/SpringOriginSSOPluginFilter.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.origin.filter.spring; + +import com.webank.wedatasphere.dss.standard.app.sso.origin.plugin.OriginSSOPluginFilter; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.filter.UserInterceptor; + +import javax.servlet.FilterConfig; +import org.springframework.stereotype.Component; +import org.springframework.web.context.WebApplicationContext; +import org.springframework.web.context.support.WebApplicationContextUtils; + + +@Component +public class SpringOriginSSOPluginFilter extends OriginSSOPluginFilter { + + @Override + public UserInterceptor getUserInterceptor(FilterConfig filterConfig) { + WebApplicationContext webApplicationContext = + WebApplicationContextUtils.getRequiredWebApplicationContext(filterConfig.getServletContext()); + return webApplicationContext.getBean(UserInterceptor.class); + } +} diff --git a/dss-standard/sso-standard/sso-integration-standard/pom.xml b/dss-standard/sso-standard/sso-integration-standard/pom.xml new file mode 100644 index 000000000..4f8d0a2e8 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/pom.xml @@ -0,0 +1,83 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-sso-integration-standard + + + + javax.servlet + javax.servlet-api + 3.1.0 + + + com.webank.wedatasphere.dss + dss-standard-common + ${dss.version} + + + + com.webank.wedatasphere.dss + dss-common + ${dss.version} + + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + scala-compile-first + + + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + + + \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/SSOIntegrationStandard.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/SSOIntegrationStandard.java new file mode 100644 index 000000000..3af53ad72 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/SSOIntegrationStandard.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOBuilderService; +import com.webank.wedatasphere.dss.standard.app.sso.builder.impl.SSOBuilderServiceImplImpl; +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.SSOPluginService; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; + + +public interface SSOIntegrationStandard extends AppStandard { + + default SSOBuilderService getSSOBuilderService() { + return SSOBuilderServiceImplImpl.getSSOBuilderService(); + } + + SSORequestService getSSORequestService(); + + SSOPluginService getSSOPluginService(); + + @Override + default String getStandardName() { + return "ssoIntegrationStandard"; + } + + @Override + default int getGrade() { + return 1; + } + + @Override + default boolean isNecessary() { + return true; + } +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/SSOIntegrationStandardFactory.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/SSOIntegrationStandardFactory.java new file mode 100644 index 000000000..f1be215bc --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/SSOIntegrationStandardFactory.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso; + +import java.io.Serializable; + +public interface SSOIntegrationStandardFactory extends Serializable { + + void init(); + + SSOIntegrationStandard getSSOIntegrationStandard(); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/Workspace.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/Workspace.java new file mode 100644 index 000000000..5f96bb28a --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/Workspace.java @@ -0,0 +1,64 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso; + +import com.webank.wedatasphere.dss.common.entity.DSSWorkspace; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.impl.SSOUrlBuilderOperationImpl; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.SSOIntegrationConf; + + +public class Workspace implements DSSWorkspace { + + protected String workspaceName; + + protected transient SSOUrlBuilderOperation ssoUrlBuilderOperation; + + private String operationStr; + + @Override + public String getWorkspaceName() { + return this.workspaceName; + } + + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + + public SSOUrlBuilderOperation getSSOUrlBuilderOperation() { + if (this.ssoUrlBuilderOperation == null) { + this.ssoUrlBuilderOperation = SSOUrlBuilderOperationImpl.restore(operationStr); + } + return this.ssoUrlBuilderOperation; + } + + + public void setSSOUrlBuilderOperation(SSOUrlBuilderOperation ssoUrlBuilderOperation) { + this.ssoUrlBuilderOperation = ssoUrlBuilderOperation; + this.operationStr = SSOIntegrationConf.gson().toJson(ssoUrlBuilderOperation); + } + + public String getOperationStr() { + return operationStr; + } + + public void setOperationStr(String operationStr) { + this.operationStr = operationStr; + } +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/DssMsgBuilderOperation.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/DssMsgBuilderOperation.java new file mode 100644 index 000000000..570d6bd81 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/DssMsgBuilderOperation.java @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder; + +import com.webank.wedatasphere.dss.standard.common.service.Operation; + +import java.util.Map; + + +public interface DssMsgBuilderOperation extends Operation { + + DssMsgBuilderOperation setQueryString(String queryString); + + DssMsgBuilderOperation setParameterMap(Map parameterMap); + + boolean isDSSMsgRequest(); + + DSSMsg getBuiltMsg(); + + interface DSSMsg { + + String getRedirectUrl(); + + String getWorkspaceName(); + + String getDSSUrl(); + + String getAppName(); + + Map getCookies(); + } + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/SSOBuilderService.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/SSOBuilderService.java new file mode 100644 index 000000000..0b9268b02 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/SSOBuilderService.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder; + + +import com.webank.wedatasphere.dss.standard.common.service.AppService; + + +public interface SSOBuilderService extends AppService { + + SSOUrlBuilderOperation createSSOUrlBuilderOperation(); + + DssMsgBuilderOperation createDssMsgBuilderOperation(); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/SSOUrlBuilderOperation.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/SSOUrlBuilderOperation.java new file mode 100644 index 000000000..f45e5ca15 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/SSOUrlBuilderOperation.java @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder; + + +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import com.webank.wedatasphere.dss.standard.common.service.Operation; + + +public interface SSOUrlBuilderOperation extends Operation { + + SSOUrlBuilderOperation setReqUrl(String reqUrl); + + SSOUrlBuilderOperation setWorkspace(String workspaceName); + + SSOUrlBuilderOperation setDSSUrl(String dssUrl); + + SSOUrlBuilderOperation setAppName(String appConnName); + + SSOUrlBuilderOperation addCookie(String key, String value); + + SSOUrlBuilderOperation redirectTo(String redirectUrl); + + SSOUrlBuilderOperation copy(); + + String getBuiltUrl() throws AppStandardErrorException; +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/DSSMsgImpl.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/DSSMsgImpl.java new file mode 100644 index 000000000..b7ccf84fb --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/DSSMsgImpl.java @@ -0,0 +1,75 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder.impl; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg; +import java.util.Map; + + +public class DSSMsgImpl implements DSSMsg { + + private String redirectUrl; + private String workspaceName; + private String dssUrl; + private String appName; + private Map cookies; + + @Override + public String getRedirectUrl() { + return redirectUrl; + } + + @Override + public String getWorkspaceName() { + return workspaceName; + } + + @Override + public String getDSSUrl() { + return dssUrl; + } + + @Override + public Map getCookies() { + return cookies; + } + + public void setRedirectUrl(String redirectUrl) { + this.redirectUrl = redirectUrl; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public void setDSSUrl(String dssUrl) { + this.dssUrl = dssUrl; + } + + @Override + public String getAppName() { + return appName; + } + + public void setAppName(String appName) { + this.appName = appName; + } + + public void setCookies(Map cookies) { + this.cookies = cookies; + } +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/DssMsgBuilderOperationImpl.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/DssMsgBuilderOperationImpl.java new file mode 100644 index 000000000..c12cce70f --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/DssMsgBuilderOperationImpl.java @@ -0,0 +1,83 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder.impl; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation; +import org.apache.commons.lang.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; + + +public class DssMsgBuilderOperationImpl implements DssMsgBuilderOperation { + + private static final Logger LOGGER = LoggerFactory.getLogger(DssMsgBuilderOperationImpl.class); + + private String queryString; + private Map parameterMap; + + @Override + public DssMsgBuilderOperation setQueryString(String queryString) { + this.queryString = queryString; + return this; + } + + @Override + public DssMsgBuilderOperation setParameterMap(Map parameterMap) { + this.parameterMap = parameterMap; + return this; + } + + @Override + public boolean isDSSMsgRequest() { + return parameterMap.containsKey("dssurl"); + } + + private String getOrNull(String key) { + String[] values = parameterMap.get(key); + if(values == null || values.length == 0) { + return null; + } else { + return values[0]; + } + } + + @Override + public DSSMsg getBuiltMsg() { + DSSMsgImpl dssMsg = new DSSMsgImpl(); + dssMsg.setRedirectUrl(getOrNull("redirect")); + dssMsg.setDSSUrl(getOrNull("dssurl")); + dssMsg.setWorkspaceName(getOrNull("workspace")); + dssMsg.setAppName(getOrNull("appName")); + Map cookies = new HashMap<>(); + String cookiesStr = getOrNull("cookies"); + if(StringUtils.isNotBlank(cookiesStr)) { + Arrays.stream(cookiesStr.split(";")).forEach(cookie -> { + int index = cookie.indexOf('='); + String key = cookie.substring(0, index).trim(); + String value = cookie.substring(index + 1).trim(); + cookies.put(key, value); + }); + } + LOGGER.info("Set cookies from dssMsg: "+cookies.toString()); + dssMsg.setCookies(cookies); + return dssMsg; + } +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/SSOBuilderServiceImplImpl.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/SSOBuilderServiceImplImpl.java new file mode 100644 index 000000000..10ed2d681 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/SSOBuilderServiceImplImpl.java @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder.impl; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOBuilderService; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.common.service.AppServiceImpl; + + +public class SSOBuilderServiceImplImpl extends AppServiceImpl implements SSOBuilderService { + + private static SSOBuilderService ssoBuilderService; + + public static SSOBuilderService getSSOBuilderService() { + if(ssoBuilderService == null) { + synchronized (SSOBuilderServiceImplImpl.class) { + if(ssoBuilderService == null) { + ssoBuilderService = new SSOBuilderServiceImplImpl(); + } + } + } + return ssoBuilderService; + } + + @Override + public SSOUrlBuilderOperation createSSOUrlBuilderOperation() { + return new SSOUrlBuilderOperationImpl(); + } + + @Override + public DssMsgBuilderOperation createDssMsgBuilderOperation() { + return new DssMsgBuilderOperationImpl(); + } + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/SSOUrlBuilderOperationImpl.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/SSOUrlBuilderOperationImpl.java new file mode 100644 index 000000000..f05e81969 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/builder/impl/SSOUrlBuilderOperationImpl.java @@ -0,0 +1,147 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.builder.impl; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.SSOIntegrationConf; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import com.webank.wedatasphere.linkis.common.conf.CommonVars; +import org.apache.commons.lang.StringUtils; + +import java.io.UnsupportedEncodingException; +import java.net.URLEncoder; +import java.util.HashMap; +import java.util.Map; +import java.util.stream.Collectors; + + +public class SSOUrlBuilderOperationImpl implements SSOUrlBuilderOperation { + + private static final String SSO_URL_FORMAT = "%s?dssurl=%s&cookies=%s&workspace=%s&appName=%s"; + private static final String SSO_REDIRECT_URL_FORMAT = "%s?redirect=%s&dssurl=%s&cookies=%s&workspace=%s&appName=%s"; + private String workspaceName; + private Map cookies = new HashMap<>(); + private String dssUrl; + private String redirectUrl; + private String reqUrl; + private String appName; + + protected String urlEncode(String str) throws AppStandardErrorException { + try { + return URLEncoder.encode(str, "utf-8"); + } catch (UnsupportedEncodingException e) { + throw new AppStandardErrorException(60001, "Encode string failed! string: " + str, e); + } + } + + @Override + public SSOUrlBuilderOperation setWorkspace(String workspaceName) { + this.workspaceName = workspaceName; + return this; + } + + @Override + public SSOUrlBuilderOperation addCookie(String key, String value) { + cookies.put(key, value); + return this; + } + + @Override + public SSOUrlBuilderOperation redirectTo(String redirectUrl) { + this.redirectUrl = redirectUrl; + return this; + } + + @Override + public SSOUrlBuilderOperation setReqUrl(String reqUrl) { + this.reqUrl = reqUrl; + return this; + } + + @Override + public SSOUrlBuilderOperation setDSSUrl(String dssUrl) { + this.dssUrl = dssUrl; + return this; + } + + @Override + public SSOUrlBuilderOperation setAppName(String appName) { + this.appName = appName; + return this; + } + + @Override + public String getBuiltUrl() throws AppStandardErrorException { + String cookieStr = cookies.entrySet().stream().map(entry -> entry.getKey() + "=" + entry.getValue()) + .collect(Collectors.joining(";")); + if(StringUtils.isNotBlank(redirectUrl)) { + return String.format(SSO_REDIRECT_URL_FORMAT, reqUrl, redirectUrl, urlEncode(dssUrl), + urlEncode(cookieStr), workspaceName, appName); + } else { + return String.format(SSO_URL_FORMAT, reqUrl, urlEncode(dssUrl), + urlEncode(cookieStr), workspaceName, appName); + } + } + + public String getWorkspaceName() { + return workspaceName; + } + + public Map getCookies() { + return cookies; + } + + public String getDssUrl() { + return dssUrl; + } + + public String getRedirectUrl() { + return redirectUrl; + } + + public String getReqUrl() { + return reqUrl; + } + + public String getAppName() { + return appName; + } + + @Override + public SSOUrlBuilderOperation copy() { + SSOUrlBuilderOperationImpl operation = new SSOUrlBuilderOperationImpl(); + operation.appName = appName; + operation.cookies = cookies; + operation.dssUrl = dssUrl; + operation.redirectUrl = redirectUrl; + operation.reqUrl = reqUrl; + operation.workspaceName = workspaceName; + return operation; + } + + public static SSOUrlBuilderOperation restore(String operationStr){ + Map operationMap = SSOIntegrationConf.gson().fromJson(operationStr, Map.class); + SSOUrlBuilderOperationImpl operation = new SSOUrlBuilderOperationImpl(); + if(operationMap.get("appName") != null) operation.appName = operationMap.get("appName").toString(); + if(operationMap.get("cookies") != null) operation.cookies = (Map) operationMap.get("cookies"); + if(operationMap.get("dssUrl") != null) operation.dssUrl = operationMap.get("dssUrl").toString(); + if(operationMap.get("redirectUrl") != null) operation.redirectUrl = operationMap.get("redirectUrl").toString(); + if(operationMap.get("reqUrl") != null) operation.reqUrl = operationMap.get("reqUrl").toString(); + if(operationMap.get("workspaceName") != null) operation.workspaceName = operationMap.get("workspaceName").toString(); + return operation; + } +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/AbstractSSOMsgParseOperation.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/AbstractSSOMsgParseOperation.java new file mode 100644 index 000000000..58776b418 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/AbstractSSOMsgParseOperation.java @@ -0,0 +1,65 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOBuilderService; +import com.webank.wedatasphere.dss.standard.app.sso.plugin.impl.SSOMsgImpl; +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg; +import javax.servlet.http.HttpServletRequest; + + +public abstract class AbstractSSOMsgParseOperation implements SSOMsgParseOperation { + + protected SSOBuilderService ssoBuilderService; + + public void setSSOBuilderService(SSOBuilderService ssoBuilderService) { + this.ssoBuilderService = ssoBuilderService; + } + + @Override + public boolean isDssRequest(HttpServletRequest request) { + return ssoBuilderService.createDssMsgBuilderOperation().setParameterMap(request.getParameterMap()).isDSSMsgRequest(); + } + + @Override + public DSSMsg getDSSMsg(HttpServletRequest request) { + return ssoBuilderService.createDssMsgBuilderOperation() + .setParameterMap(request.getParameterMap()).getBuiltMsg(); + } + + @Override + public SSOMsg getSSOMsg(HttpServletRequest request) { + DSSMsg dssMsg = getDSSMsg(request); + SSOMsg ssoMsg = createSSOMsg(); + String user = getUser(dssMsg); + setSSOMsg(ssoMsg, dssMsg, user); + return ssoMsg; + } + + protected SSOMsg createSSOMsg() { + return new SSOMsgImpl(); + } + + protected void setSSOMsg(SSOMsg ssoMsg, DSSMsg dssMsg, String user) { + SSOMsgImpl ssoMsgImpl = (SSOMsgImpl) ssoMsg; + ssoMsgImpl.setRedirectUrl(dssMsg.getRedirectUrl()); + ssoMsgImpl.setWorkspaceName(dssMsg.getWorkspaceName()); + ssoMsgImpl.setUser(user); + } + + protected abstract String getUser(DSSMsg dssMsg); +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/AbstractWorkspacePlugin.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/AbstractWorkspacePlugin.java new file mode 100644 index 000000000..f7ea6fee6 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/AbstractWorkspacePlugin.java @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg; +import java.util.List; +import javax.servlet.http.HttpServletRequest; + + +public abstract class AbstractWorkspacePlugin implements WorkspacePlugin { + + private DssMsgCacheOperation dssMsgCacheOperation; + + public void setDssMsgCacheOperation(DssMsgCacheOperation dssMsgCacheOperation) { + this.dssMsgCacheOperation = dssMsgCacheOperation; + } + + @Override + public String getWorkspaceName(HttpServletRequest req) { + return dssMsgCacheOperation.getWorkspaceInSession(req); + } + + @Override + public List getAllUsers(HttpServletRequest req) { + DSSMsg dssMsg = dssMsgCacheOperation.getDSSMsgInSession(req); + return getAllUsers(dssMsg); + } + + protected abstract List getAllUsers(DSSMsg dssMsg); +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/DssMsgCacheOperation.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/DssMsgCacheOperation.java new file mode 100644 index 000000000..04fdf906f --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/DssMsgCacheOperation.java @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + +import com.webank.wedatasphere.dss.standard.common.service.Operation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg; +import javax.servlet.http.HttpServletRequest; + + +public interface DssMsgCacheOperation extends Operation { + + void setWorkspaceToSession(HttpServletRequest req, String workspaceName); + + String getWorkspaceInSession(HttpServletRequest req); + + DSSMsg getDSSMsgInSession(HttpServletRequest request); + + void setDSSMsgToSession(DSSMsg dssMsg, HttpServletRequest request); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOMsg.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOMsg.java new file mode 100644 index 000000000..4b2dac42b --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOMsg.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + + +public interface SSOMsg { + + String getRedirectUrl(); + + String getWorkspaceName(); + + String getUser(); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOMsgParseOperation.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOMsgParseOperation.java new file mode 100644 index 000000000..88500b867 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOMsgParseOperation.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + +import com.webank.wedatasphere.dss.standard.common.service.Operation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg; +import javax.servlet.http.HttpServletRequest; + + +public interface SSOMsgParseOperation extends Operation { + + boolean isDssRequest(HttpServletRequest request); + + DSSMsg getDSSMsg(HttpServletRequest request); + + SSOMsg getSSOMsg(HttpServletRequest request); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOPluginService.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOPluginService.java new file mode 100644 index 000000000..df1344ec3 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOPluginService.java @@ -0,0 +1,36 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOBuilderService; +import com.webank.wedatasphere.dss.standard.common.service.AppService; +import java.io.Closeable; + + +public interface SSOPluginService extends AppService, Closeable { + + void setSSOBuilderService(SSOBuilderService ssoBuilderService); + + SSOMsgParseOperation createSSOMsgParseOperation(); + + WorkspacePlugin createWorkspacePluginOperation(); + + default DssMsgCacheOperation createDssMsgCacheOperation() { + return DssMsgCacheOperationImpl.getDssMsgCacheOperation(); + } + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/WorkspacePlugin.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/WorkspacePlugin.java new file mode 100644 index 000000000..844ff4274 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/WorkspacePlugin.java @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin; + +import com.webank.wedatasphere.dss.standard.common.service.Operation; + +import java.util.List; +import javax.servlet.http.HttpServletRequest; + + +public interface WorkspacePlugin extends Operation { + + List getAllUsers(HttpServletRequest req); + + String getWorkspaceName(HttpServletRequest req); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/DSSInternalUserInterceptor.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/DSSInternalUserInterceptor.java new file mode 100644 index 000000000..8deca7f10 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/DSSInternalUserInterceptor.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin.filter; + +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg; +import javax.servlet.http.HttpServletRequest; + + +public interface DSSInternalUserInterceptor extends UserInterceptor { + + HttpServletRequest addCookiesToRequest(DSSMsg dssMsg, HttpServletRequest req); + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/HttpRequestUserInterceptor.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/HttpRequestUserInterceptor.java new file mode 100644 index 000000000..cb1a8f235 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/HttpRequestUserInterceptor.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin.filter; + +import javax.servlet.http.HttpServletRequest; + + +public interface HttpRequestUserInterceptor extends UserInterceptor { + + HttpServletRequest addUserToRequest(String user, HttpServletRequest req); + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/HttpSessionUserInterceptor.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/HttpSessionUserInterceptor.java new file mode 100644 index 000000000..fa7038531 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/HttpSessionUserInterceptor.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin.filter; + +import javax.servlet.http.HttpServletRequest; + + +public interface HttpSessionUserInterceptor extends UserInterceptor { + + void addUserToSession(String user, HttpServletRequest req); + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/SSOPluginFilter.scala b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/SSOPluginFilter.scala new file mode 100644 index 000000000..6f24b9a7a --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/SSOPluginFilter.scala @@ -0,0 +1,82 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin.filter + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard +import javax.servlet._ +import javax.servlet.http.{HttpServletRequest, HttpServletResponse} +import org.apache.commons.lang.StringUtils + + +abstract class SSOPluginFilter extends Filter { + + private var userInterceptor: UserInterceptor = _ + + override def init(filterConfig: FilterConfig): Unit = { + userInterceptor = getUserInterceptor(filterConfig) + getSSOIntegrationStandard.init() + } + + def info(str: String): Unit + + protected def getUserInterceptor(filterConfig: FilterConfig): UserInterceptor + + protected def getSSOIntegrationStandard: SSOIntegrationStandard + + override def doFilter(servletRequest: ServletRequest, servletResponse: ServletResponse, + filterChain: FilterChain): Unit = { + val req = servletRequest.asInstanceOf[HttpServletRequest] + val resp = servletResponse.asInstanceOf[HttpServletResponse] + val ssoPluginService = getSSOIntegrationStandard.getSSOPluginService + val dssMsg = ssoPluginService.createSSOMsgParseOperation().getDSSMsg(req) + if (ssoPluginService.createSSOMsgParseOperation.isDssRequest(req)) { + val (redirectUrl, workspaceName, wrappedReq) = if(!userInterceptor.isUserExistInSession(req)) { + val ssoMsg = ssoPluginService.createSSOMsgParseOperation.getSSOMsg(req) + val username = ssoMsg.getUser + val wrappedReq = userInterceptor match { + case interceptor: HttpSessionUserInterceptor => + interceptor.addUserToSession(username, req) + req + case interceptor: HttpRequestUserInterceptor => interceptor.addUserToRequest(username, req) + case interceptor: DSSInternalUserInterceptor => interceptor.addCookiesToRequest(dssMsg, req) + } + if(wrappedReq != req) { + wrappedReq.getCookies.foreach(resp.addCookie) + } + info(s"DSS User: $username succeed to login. and wrappedReq cookies is "+wrappedReq.getCookies.toString) + (ssoMsg.getRedirectUrl, ssoMsg.getWorkspaceName, wrappedReq) + } else { + (dssMsg.getRedirectUrl, dssMsg.getWorkspaceName, req) + } + val workspaceOperation = ssoPluginService.createDssMsgCacheOperation() + if(workspaceName != workspaceOperation.getWorkspaceInSession(req)) { + info(s"Set DSS workspace to: $workspaceName.") + workspaceOperation.setWorkspaceToSession(req, workspaceName) + } + if(workspaceOperation.getDSSMsgInSession(req) == null) { + workspaceOperation.setDSSMsgToSession(dssMsg, req) + } + if(StringUtils.isNotBlank(redirectUrl)) resp.sendRedirect(redirectUrl) + else filterChain.doFilter(wrappedReq, servletResponse) + } else { + filterChain.doFilter(servletRequest, servletResponse) + } + } + + override def destroy(): Unit = {} + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/UserInterceptor.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/UserInterceptor.java new file mode 100644 index 000000000..acf745682 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/filter/UserInterceptor.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin.filter; + +import javax.servlet.http.HttpServletRequest; + + +public interface UserInterceptor { + + boolean isUserExistInSession(HttpServletRequest req); + + String getUser(HttpServletRequest req); + +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/impl/SSOMsgImpl.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/impl/SSOMsgImpl.java new file mode 100644 index 000000000..05da2833a --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/plugin/impl/SSOMsgImpl.java @@ -0,0 +1,54 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin.impl; + +import com.webank.wedatasphere.dss.standard.app.sso.plugin.SSOMsg; + + +public class SSOMsgImpl implements SSOMsg { + + private String redirectUrl; + private String user; + private String workspaceName; + + @Override + public String getRedirectUrl() { + return redirectUrl; + } + + public void setRedirectUrl(String redirectUrl) { + this.redirectUrl = redirectUrl; + } + + @Override + public String getUser() { + return user; + } + + public void setUser(String user) { + this.user = user; + } + + @Override + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/request/SSORequestOperation.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/request/SSORequestOperation.java new file mode 100644 index 000000000..5a46a9131 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/request/SSORequestOperation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.request; + +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardErrorException; +import com.webank.wedatasphere.dss.standard.common.service.Operation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; + + +public interface SSORequestOperation extends Operation { + + R requestWithSSO(SSOUrlBuilderOperation urlBuilder, T req) throws AppStandardErrorException; + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/request/SSORequestService.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/request/SSORequestService.java new file mode 100644 index 000000000..dfdbf852d --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/sso/request/SSORequestService.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.request; + + +import com.webank.wedatasphere.dss.standard.common.service.AppService; + + +public interface SSORequestService extends AppService { + + SSORequestOperation createSSORequestOperation(String appName); + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/sso/utils/SSOHelper.java b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/sso/utils/SSOHelper.java new file mode 100644 index 000000000..d6a583fee --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/sso/utils/SSOHelper.java @@ -0,0 +1,46 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.sso.utils; + +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.app.sso.builder.SSOUrlBuilderOperation; +import com.webank.wedatasphere.dss.standard.app.sso.builder.impl.SSOUrlBuilderOperationImpl; +import com.webank.wedatasphere.linkis.common.conf.Configuration; +import java.util.Arrays; +import javax.servlet.http.Cookie; +import javax.servlet.http.HttpServletRequest; + + +public class SSOHelper { + + public static Workspace getWorkspace(HttpServletRequest request){ + Cookie[] cookies = request.getCookies(); + Cookie workspaceCookie = Arrays.stream(cookies). + filter(cookie -> "workspaceId".equals(cookie.getName())).findAny().orElse(null); + Workspace workspace = new Workspace(); + if (workspaceCookie != null) { + workspace.setWorkspaceName(workspaceCookie.getValue()); + } + SSOUrlBuilderOperation ssoUrlBuilderOperation = new SSOUrlBuilderOperationImpl(); + Arrays.stream(cookies).forEach(cookie -> ssoUrlBuilderOperation.addCookie(cookie.getName(), cookie.getValue())); + ssoUrlBuilderOperation.setDSSUrl(Configuration.GATEWAY_URL().getValue()); + ssoUrlBuilderOperation.setWorkspace(workspace.getWorkspaceName()); + workspace.setSSOUrlBuilderOperation(ssoUrlBuilderOperation); + return workspace; + } + +} diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/plugin/DssMsgCacheOperationImpl.scala b/dss-standard/sso-standard/sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/plugin/DssMsgCacheOperationImpl.scala new file mode 100644 index 000000000..f6d941fb2 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/plugin/DssMsgCacheOperationImpl.scala @@ -0,0 +1,70 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin + +import java.lang +import java.lang.reflect.Type + +import com.google.gson.{GsonBuilder, JsonElement, JsonPrimitive, JsonSerializationContext, JsonSerializer} +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg +import com.webank.wedatasphere.dss.standard.app.sso.builder.impl.DSSMsgImpl +import javax.servlet.http.HttpServletRequest + + +class DssMsgCacheOperationImpl private() extends DssMsgCacheOperation { + + override def setWorkspaceToSession(req: HttpServletRequest, workspaceName: String): Unit = + getDSSMsgInSession(req) match { + case dssMsg: DSSMsgImpl => dssMsg.setWorkspaceName(workspaceName) + case _ => + } + + override def getWorkspaceInSession(req: HttpServletRequest): String = + getDSSMsgInSession(req) match { + case dssMsg: DSSMsgImpl => dssMsg.getWorkspaceName + case _ => null + } + + override def getDSSMsgInSession(request: HttpServletRequest): DssMsgBuilderOperation.DSSMsg = + request.getSession.getAttribute(DssMsgCacheOperationImpl.DSS_MSG_KEY) match { + case dssMsg: DSSMsgImpl => dssMsg + case _ => null + } + + override def setDSSMsgToSession(dssMsg: DssMsgBuilderOperation.DSSMsg, request: HttpServletRequest): Unit = { + val newDSSMsg = dssMsg match { + case dss: DSSMsgImpl => + dss.setRedirectUrl(null) + dss + case dss: DSSMsg => + val dssMsg1 = new DSSMsgImpl + dssMsg1.setCookies(dss.getCookies) + dssMsg1.setDSSUrl(dss.getDSSUrl) + dssMsg1.setWorkspaceName(dss.getWorkspaceName) + dssMsg1 + } + request.getSession.setAttribute(DssMsgCacheOperationImpl.DSS_MSG_KEY, newDSSMsg) + } +} +object DssMsgCacheOperationImpl { + private val DSS_MSG_KEY = "dss_msg_key" + + private val dssMsgCacheOperation = new DssMsgCacheOperationImpl + + def getDssMsgCacheOperation: DssMsgCacheOperation = dssMsgCacheOperation +} \ No newline at end of file diff --git a/dss-standard/sso-standard/sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOIntegrationConf.scala b/dss-standard/sso-standard/sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOIntegrationConf.scala new file mode 100644 index 000000000..8f90830e6 --- /dev/null +++ b/dss-standard/sso-standard/sso-integration-standard/src/main/scala/com/webank/wedatasphere/dss/standard/app/sso/plugin/SSOIntegrationConf.scala @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.sso.plugin + +import java.lang +import java.lang.reflect.Type + +import com.google.gson.{GsonBuilder, JsonElement, JsonPrimitive, JsonSerializationContext, JsonSerializer} + + +object SSOIntegrationConf { + implicit val gson = new GsonBuilder().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").serializeNulls + .registerTypeAdapter(classOf[java.lang.Double], new JsonSerializer[java.lang.Double] { + override def serialize(t: lang.Double, `type`: Type, jsonSerializationContext: JsonSerializationContext): JsonElement = + if(t == t.longValue()) new JsonPrimitive(t.longValue()) else new JsonPrimitive(t) + }).create +} diff --git a/dss-standard/structure-standard/dss-project-plugin/pom.xml b/dss-standard/structure-standard/dss-project-plugin/pom.xml new file mode 100644 index 000000000..85cf9febb --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/pom.xml @@ -0,0 +1,67 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-project-plugin + + + + com.webank.wedatasphere.dss + dss-origin-sso-integration-standard + ${dss.version} + compile + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + + + \ No newline at end of file diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectAuth.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectAuth.java new file mode 100644 index 000000000..fcb0f41ac --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectAuth.java @@ -0,0 +1,40 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin; + +import java.util.List; + + +public interface ProjectAuth { + + String getWorkspaceName(); + + String getProjectId(); + + String getProjectName(); + + List getEditUsers(); + + List getAccessUsers(); + + List getExecuteUsers(); + + default List getDeleteUsers() { + return getEditUsers(); + } + +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectCooperationPlugin.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectCooperationPlugin.java new file mode 100644 index 000000000..aff15ef1e --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectCooperationPlugin.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin; + +import javax.servlet.http.HttpServletRequest; + + +public interface ProjectCooperationPlugin { + + ProjectAuth getProjectAuth(HttpServletRequest request, String projectId); + + ProjectAuth getProjectAuthByName(HttpServletRequest request, String projectName); + +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectPlugin.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectPlugin.java new file mode 100644 index 000000000..4f7e64900 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/ProjectPlugin.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin; + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard; +import com.webank.wedatasphere.dss.standard.app.sso.origin.OriginSSOIntegrationStandardFactory; + +import javax.servlet.http.HttpServletRequest; +import java.util.ArrayList; +import java.util.List; +import java.util.function.Function; +import java.util.stream.Collectors; + + +public interface ProjectPlugin { + + //TODO 如果一个用户从dss空间进入到qualitis建立工程，添加一个是否是创建工程的请求，添加一个开关参数，是否允许用户建工程 + List getProjects(HttpServletRequest request); + + default List filterProjects(List projects, HttpServletRequest request, + Function getProjectId) { + SSOIntegrationStandard ssoIntegrationStandard = new OriginSSOIntegrationStandardFactory().getSSOIntegrationStandard(); + if(!ssoIntegrationStandard.getSSOPluginService().createSSOMsgParseOperation().isDssRequest(request)) { + return projects; + } + List projectList = getProjects(request); + if(projectList == null) { + return new ArrayList<>(); + } + List filteredList = projects.stream().filter(t -> projectList.contains(getProjectId.apply(t))) + .collect(Collectors.toList()); + return filteredList; + } + +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/AbstractProjectAuthInterceptor.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/AbstractProjectAuthInterceptor.java new file mode 100644 index 000000000..10ca96d68 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/AbstractProjectAuthInterceptor.java @@ -0,0 +1,61 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.filter; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.ProjectAuth; +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient; +import com.webank.wedatasphere.dss.standard.common.exception.AppStandardWarnException; + +import java.util.ArrayList; +import java.util.List; +import javax.servlet.http.HttpServletRequest; + + +public abstract class AbstractProjectAuthInterceptor implements ProjectAuthInterceptor { + + private List projectUris = new ArrayList<>(); + + @Override + public boolean isProjectRequest(HttpServletRequest request) { + String requestUri = request.getRequestURI(); + return projectUris.stream().anyMatch(requestUri::matches); + } + + protected void addProjectUri(String projectUri) { + projectUris.add(projectUri); + } + + @Override + public String getForbiddenMsg(ProjectAuth projectAuth, ProjectRequestType projectRequestType, + HttpServletRequest request) { + String message = "You have no permission to " + projectRequestType.toString().toLowerCase() + + " the content of project " + projectAuth.getProjectName(); + Object returnObj = getForbiddenMsg(message); + if(returnObj instanceof String) { + return (String) returnObj; + } else { + try { + return HttpClient.objectMapper().writeValueAsString(returnObj); + } catch (JsonProcessingException e) { + throw new AppStandardWarnException(50002, "Serialize object to string failed! Object: " + returnObj, e); + } + } + } + + protected abstract Object getForbiddenMsg(String message); +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectAuthInterceptor.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectAuthInterceptor.java new file mode 100644 index 000000000..d17469ef2 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectAuthInterceptor.java @@ -0,0 +1,37 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.filter; + +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.ProjectAuth; + +import javax.servlet.http.HttpServletRequest; + + +public interface ProjectAuthInterceptor { + + boolean isProjectRequest(HttpServletRequest request); + + String getProjectId(HttpServletRequest request); + + String getProjectName(HttpServletRequest request); + + ProjectRequestType getProjectRequestType(HttpServletRequest request); + + String getForbiddenMsg(ProjectAuth projectAuth, ProjectRequestType projectRequestType, + HttpServletRequest request); + +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectRequestType.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectRequestType.java new file mode 100644 index 000000000..3c34346d4 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectRequestType.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.filter; + + +public enum ProjectRequestType { + Edit, Access, Delete, Execute +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthImpl.java b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthImpl.java new file mode 100644 index 000000000..062deea85 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthImpl.java @@ -0,0 +1,111 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin; + +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.ProjectAuth; + +import java.util.List; + + +public class ProjectAuthImpl implements ProjectAuth { + + private String workspaceName; + private String projectId; + private String projectName; + private List editUsers; + private List accessUsers; + private List deleteUsers; + private List executeUsers; + + @Override + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + @Override + public String getProjectId() { + return projectId; + } + + public void setProjectId(String projectId) { + this.projectId = projectId; + } + + @Override + public String getProjectName() { + return projectName; + } + + public void setProjectName(String projectName) { + this.projectName = projectName; + } + + @Override + public List getEditUsers() { + return editUsers; + } + + public void setEditUsers(List editUsers) { + this.editUsers = editUsers; + } + + @Override + public List getAccessUsers() { + return accessUsers; + } + + @Override + public List getExecuteUsers() { + return executeUsers; + } + + public void setExecuteUsers(List executeUsers) { + this.executeUsers = executeUsers; + } + + public void setAccessUsers(List accessUsers) { + this.accessUsers = accessUsers; + } + + @Override + public List getDeleteUsers() { + if(deleteUsers == null || deleteUsers.isEmpty()) { + return editUsers; + } + return deleteUsers; + } + + public void setDeleteUsers(List deleteUsers) { + this.deleteUsers = deleteUsers; + } + + @Override + public String toString() { + return "ProjectAuthImpl(" + + "workspaceName='" + workspaceName + '\'' + + ", projectId='" + projectId + '\'' + + ", projectName='" + projectName + '\'' + + ", editUsers=" + editUsers + + ", accessUsers=" + accessUsers + + ", deleteUsers=" + deleteUsers + + ')'; + } +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/conf/ProjectCooperateConfiguration.scala b/dss-standard/structure-standard/dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/conf/ProjectCooperateConfiguration.scala new file mode 100644 index 000000000..34f1a4c81 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/conf/ProjectCooperateConfiguration.scala @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.conf + +import com.webank.wedatasphere.linkis.common.conf.{CommonVars, TimeType} + + +object ProjectCooperateConfiguration { + + val EXPIRE_PROJECT_AUTH_TIMEOUT = CommonVars("wds.dss.project.auth.timeout", new TimeType("10m")) + val EXPIRE_PROJECT_AUTH_SCAN_INTERVAL = CommonVars("wds.dss.project.auth.scan.interval", new TimeType("5m")) + +} diff --git a/dss-standard/structure-standard/dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectCooperationFilter.scala b/dss-standard/structure-standard/dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectCooperationFilter.scala new file mode 100644 index 000000000..13aab22f6 --- /dev/null +++ b/dss-standard/structure-standard/dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectCooperationFilter.scala @@ -0,0 +1,167 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.filter + +import java.util +import java.util.concurrent.TimeUnit + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard +import com.webank.wedatasphere.dss.standard.app.sso.origin.OriginSSOIntegrationStandardFactory +import com.webank.wedatasphere.dss.standard.app.sso.plugin.filter.UserInterceptor +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.conf.ProjectCooperateConfiguration +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.filter.ProjectRequestType.{Access, Delete, Edit, Execute} +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.{ProjectAuth, ProjectCooperationPlugin} +import com.webank.wedatasphere.linkis.common.conf.Configuration +import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} +import com.webank.wedatasphere.linkis.httpclient.exception.HttpClientResultException +import javax.servlet._ +import javax.servlet.http.{HttpServletRequest, HttpServletResponse} +import org.apache.commons.lang.StringUtils + + +abstract class ProjectCooperationFilter extends Filter with Logging { + + protected var projectAuthInterceptor: ProjectAuthInterceptor = _ + protected var userInterceptor: UserInterceptor = _ + protected var projectCooperationPlugin: ProjectCooperationPlugin = _ + private val cachedProjectAuthsWithId = new util.HashMap[String, TimeoutProjectAuth]() + private val cachedProjectAuthsWithName = new util.HashMap[String, TimeoutProjectAuth]() + private val ignoreProjectWithId = new util.ArrayList[String]() + private val ignoreProjectWithName = new util.ArrayList[String]() + + override def init(filterConfig: FilterConfig): Unit = { + projectAuthInterceptor = getProjectAuthInterceptor(filterConfig) + userInterceptor = getUserAuthInterceptor(filterConfig) + projectCooperationPlugin = getProjectCooperationPlugin(filterConfig) + } + + protected def getProjectAuthInterceptor(filterConfig: FilterConfig): ProjectAuthInterceptor + protected def getUserAuthInterceptor(filterConfig: FilterConfig): UserInterceptor + protected def getProjectCooperationPlugin(filterConfig: FilterConfig): ProjectCooperationPlugin + + override def doFilter(servletRequest: ServletRequest, servletResponse: ServletResponse, + filterChain: FilterChain): Unit = { + val req = servletRequest.asInstanceOf[HttpServletRequest] + val uri = req.getRequestURI + val ssoIntegrationStandard:SSOIntegrationStandard =new OriginSSOIntegrationStandardFactory().getSSOIntegrationStandard + if(ssoIntegrationStandard.getSSOPluginService + .createDssMsgCacheOperation().getWorkspaceInSession(req) == null) { + info(s"Request $uri has no relation with DSS, ignore it.") + filterChain.doFilter(servletRequest, servletResponse) + return + } + val resp = servletResponse.asInstanceOf[HttpServletResponse] + if(!projectAuthInterceptor.isProjectRequest(req)) { + info(s"Request $uri is not a project request, ignore it.") + filterChain.doFilter(req, resp) + return + } + val projectId = projectAuthInterceptor.getProjectId(req) + if(ignoreProjectWithId.contains(projectId)) { + info(s"The project $projectId is not a DSS project, ignore it, request is $uri.") + filterChain.doFilter(req, resp) + return + } + val projectAuth = if(StringUtils.isNotBlank(projectId)) { + getProjectAuthById(req, projectId) + } else { + val projectName = projectAuthInterceptor.getProjectName(req) + // If this request cannot get the projectId or projectName, ignore it. + if(StringUtils.isBlank(projectName) || ignoreProjectWithName.contains(projectName)) { + info(s"This request $uri cannot get the projectId or projectName, ignore it.") + filterChain.doFilter(req, resp) + return + } + getProjectAuthByName(req, projectName) + } + if(projectAuth == null) { + filterChain.doFilter(req, resp) + return + } + val projectRequestType = projectAuthInterceptor.getProjectRequestType(req) + debug(s"RequestURI: $uri, ProjectRequestType: $projectRequestType, ProjectAuth: $projectAuth.") + val user = userInterceptor.getUser(req) + val passed = projectRequestType match { + case Edit => projectAuth.getEditUsers.contains(user) + case Access => projectAuth.getAccessUsers.contains(user) + case Delete => projectAuth.getDeleteUsers.contains(user) + case Execute => projectAuth.getDeleteUsers.contains(user) + } + if(passed) { + info(s"Cooperation Project request $uri has passed auth validation.") + filterChain.doFilter(req, resp) + } else { + val msg = projectAuthInterceptor.getForbiddenMsg(projectAuth, projectRequestType, req) + info(s"Cooperation Project request $uri has been failed with auth validation.") + resp.getOutputStream.write(msg.getBytes(Configuration.BDP_ENCODING.getValue)) + resp.setStatus(403) + resp.getOutputStream.flush() + } + } + + protected def getProjectAuthById(req: HttpServletRequest, projectId: String): ProjectAuth = + getProjectAuth(projectId, cachedProjectAuthsWithId, + projectCooperationPlugin.getProjectAuth(req, projectId), ignoreProjectWithId) + + protected def getProjectAuthByName(req: HttpServletRequest, projectName: String): ProjectAuth = + getProjectAuth(projectName, cachedProjectAuthsWithName, + projectCooperationPlugin.getProjectAuthByName(req, projectName), ignoreProjectWithName) + + private def getProjectAuth(key: String, + map: util.HashMap[String, TimeoutProjectAuth], + getNewProjectAuth: => ProjectAuth, + list: util.ArrayList[String]): ProjectAuth = { + var timeoutProjectAuth = map.get(key) + if(timeoutProjectAuth == null || timeoutProjectAuth.isTimeout) { + key.intern().synchronized { + timeoutProjectAuth = map.get(key) + if(timeoutProjectAuth == null || timeoutProjectAuth.isTimeout) { + val projectAuth = Utils.tryCatch(getNewProjectAuth) { + case t: HttpClientResultException => + if(t.getDesc.contains("not exists")) { + list.add(key) + return null + } else throw t + case t: Throwable => throw t + } + map.put(key, TimeoutProjectAuth(projectAuth)) + } + } + } + map.get(key).projectAuth + } + + override def destroy(): Unit = {} + + private case class TimeoutProjectAuth(createTime: Long, projectAuth: ProjectAuth) { + def isTimeout: Boolean = System.currentTimeMillis() - createTime > ProjectCooperateConfiguration.EXPIRE_PROJECT_AUTH_TIMEOUT.getValue.toLong + } + private object TimeoutProjectAuth { + def apply(projectAuth: ProjectAuth): TimeoutProjectAuth = + new TimeoutProjectAuth(System.currentTimeMillis(), projectAuth) + } + + Utils.defaultScheduler.scheduleAtFixedRate(new Runnable { + override def run(): Unit = { + cachedProjectAuthsWithId.keySet().toArray + .foreach{ case k: String => if(cachedProjectAuthsWithId.get(k).isTimeout) cachedProjectAuthsWithId.remove(k)} + cachedProjectAuthsWithName.keySet().toArray + .foreach{ case k: String => if(cachedProjectAuthsWithName.get(k).isTimeout) cachedProjectAuthsWithName.remove(k)} + } + } , ProjectCooperateConfiguration.EXPIRE_PROJECT_AUTH_SCAN_INTERVAL.getValue.toLong, ProjectCooperateConfiguration.EXPIRE_PROJECT_AUTH_SCAN_INTERVAL.getValue.toLong, TimeUnit.MILLISECONDS) + +} diff --git a/dss-standard/structure-standard/dss-role-plugin/pom.xml b/dss-standard/structure-standard/dss-role-plugin/pom.xml new file mode 100644 index 000000000..7cc6017c0 --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/pom.xml @@ -0,0 +1,66 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-role-plugin + + + + com.webank.wedatasphere.dss + dss-origin-sso-integration-standard + ${dss.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + + + \ No newline at end of file diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/DSSUserRoles.java b/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/DSSUserRoles.java new file mode 100644 index 000000000..3294a1b76 --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/DSSUserRoles.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role; + +import java.util.List; + + +public interface DSSUserRoles { + + List getRoles(); + + String getWorkspaceName(); + + String getUser(); + +} diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/RolePlugin.java b/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/RolePlugin.java new file mode 100644 index 000000000..47ab3d26d --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/RolePlugin.java @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role; + +import com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl.OriginRolePlugin; + +import java.util.List; +import javax.servlet.http.HttpServletRequest; + + +public interface RolePlugin { + + DSSUserRoles getRoles(HttpServletRequest request); + + List getUsers(HttpServletRequest request, String role); + + static RolePlugin getRolePlugin() { + return OriginRolePlugin.getRolePlugin(); + } + +} diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/DSSUserRolesImpl.java b/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/DSSUserRolesImpl.java new file mode 100644 index 000000000..2d5e256d9 --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/DSSUserRolesImpl.java @@ -0,0 +1,56 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl; + +import com.webank.wedatasphere.dss.standard.app.structure.plugin.role.DSSUserRoles; + +import java.util.List; + + +public class DSSUserRolesImpl implements DSSUserRoles { + + private List roles; + private String workspaceName; + private String user; + + public void setRoles(List roles) { + this.roles = roles; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + public void setUser(String user) { + this.user = user; + } + + @Override + public List getRoles() { + return roles; + } + + @Override + public String getWorkspaceName() { + return workspaceName; + } + + @Override + public String getUser() { + return user; + } +} diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/OriginRolePlugin.scala b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/OriginRolePlugin.scala new file mode 100644 index 000000000..307b7ac5c --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/OriginRolePlugin.scala @@ -0,0 +1,64 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl + +import java.util + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard +import com.webank.wedatasphere.dss.standard.app.sso.origin.OriginSSOIntegrationStandardFactory +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient +import com.webank.wedatasphere.dss.standard.app.structure.plugin.role.{DSSUserRoles, RolePlugin} +import javax.servlet.http.HttpServletRequest + + +class OriginRolePlugin private() extends RolePlugin { + val ssoIntegrationStandard:SSOIntegrationStandard =new OriginSSOIntegrationStandardFactory().getSSOIntegrationStandard + + override def getRoles(request: HttpServletRequest): DSSUserRoles = { + val dssMsg = ssoIntegrationStandard.getSSOPluginService + .createDssMsgCacheOperation().getDSSMsgInSession(request) + val dwsHttpClient = HttpClient.getDSSClient(dssMsg.getDSSUrl) + val dssPrivilege = new DSSUserRolesImpl + dssPrivilege.setWorkspaceName(dssMsg.getWorkspaceName) + val userRoleInfoAction = new RoleInfoOfUserAction + userRoleInfoAction.setWorkspace(dssMsg.getWorkspaceName) + HttpClient.addCookies(dssMsg, userRoleInfoAction) + dwsHttpClient.execute(userRoleInfoAction) match { + case userRoleInfoResult: RoleInfoOfUserResult => + dssPrivilege.setRoles(userRoleInfoResult.getRoles) + dssPrivilege.setUser(userRoleInfoResult.getUser) + } + dssPrivilege + } + + override def getUsers(request: HttpServletRequest, role: String): util.List[String] = { + val dssMsg = ssoIntegrationStandard.getSSOPluginService + .createDssMsgCacheOperation().getDSSMsgInSession(request) + val dwsHttpClient = HttpClient.getDSSClient(dssMsg.getDSSUrl) + val userInfoOfRole = new UserInfoOfRoleAction + userInfoOfRole.setWorkspace(dssMsg.getWorkspaceName) + HttpClient.addCookies(dssMsg, userInfoOfRole) + dwsHttpClient.execute(userInfoOfRole) match { + case result: UserInfoOfRoleResult => + result.getUsers + } + } +} +object OriginRolePlugin { + private val rolePlugin = new OriginRolePlugin + def getRolePlugin: RolePlugin = rolePlugin +} \ No newline at end of file diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/RoleInfoOfUserAction.scala b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/RoleInfoOfUserAction.scala new file mode 100644 index 000000000..93bc6654e --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/RoleInfoOfUserAction.scala @@ -0,0 +1,31 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction + + +class RoleInfoOfUserAction extends POSTAction with DWSHttpAction { + + override def suffixURLs: Array[String] = Array("dss", "getRolesOfWorkspaceUser") + + override def getRequestPayload: String = "" + + def setWorkspace(workspace: String): Unit = addRequestPayload("workspaceName", workspace) + +} diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/RoleInfoOfUserResult.scala b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/RoleInfoOfUserResult.scala new file mode 100644 index 000000000..4f5b050a9 --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/RoleInfoOfUserResult.scala @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl + +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +import scala.beans.BeanProperty + + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/getRolesOfWorkspaceUser") +class RoleInfoOfUserResult extends DWSResult { + + @BeanProperty var user: String = _ + + @BeanProperty var roles: java.util.List[String] = _ + +} diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/UserInfoOfRoleAction.scala b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/UserInfoOfRoleAction.scala new file mode 100644 index 000000000..5b696e028 --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/UserInfoOfRoleAction.scala @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction + + +class UserInfoOfRoleAction extends POSTAction with DWSHttpAction { + + override def suffixURLs: Array[String] = Array("dss", "getUsersOfWorkspaceRole") + + def setWorkspace(workspace: String): Unit = addRequestPayload("workspaceName", workspace) + + override def getRequestPayload: String = "" +} \ No newline at end of file diff --git a/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/UserInfoOfRoleResult.scala b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/UserInfoOfRoleResult.scala new file mode 100644 index 000000000..a6b1531c2 --- /dev/null +++ b/dss-standard/structure-standard/dss-role-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/plugin/role/impl/UserInfoOfRoleResult.scala @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.plugin.role.impl + +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +import scala.beans.BeanProperty + + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/getUsersOfWorkspaceRole") +class UserInfoOfRoleResult extends DWSResult { + + @BeanProperty var users: java.util.List[String] = _ + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/pom.xml b/dss-standard/structure-standard/dss-structure-integration-standard/pom.xml new file mode 100644 index 000000000..88fe5d650 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/pom.xml @@ -0,0 +1,43 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + dss-structure-integration-standard + + + com.webank.wedatasphere.linkis + linkis-common + ${linkis.version} + + + com.webank.wedatasphere.dss + dss-sso-integration-standard + ${dss.version} + + + + \ No newline at end of file diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/AbstractStructureIntegrationStandard.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/AbstractStructureIntegrationStandard.java new file mode 100644 index 000000000..2afaffa44 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/AbstractStructureIntegrationStandard.java @@ -0,0 +1,70 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure; + +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.role.RoleService; +import com.webank.wedatasphere.dss.standard.app.structure.status.AppStatusService; +import com.webank.wedatasphere.dss.standard.common.core.AbstractAppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; + +import java.io.IOException; + +public abstract class AbstractStructureIntegrationStandard extends AbstractAppIntegrationStandard + implements StructureIntegrationStandard { + + protected abstract ProjectService createProjectService(); + + protected RoleService createRoleService() { + return null; + } + + protected AppStatusService createAppStatusService() { + return null; + } + + @Override + protected void initService(T service) { + service.setAppStandard(this); + } + + @Override + public void init() { + } + + @Override + public RoleService getRoleService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createRoleService, RoleService.class); + } + + @Override + public ProjectService getProjectService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createProjectService, ProjectService.class); + } + + @Override + public AppStatusService getAppStateService(AppInstance appInstance) { + return getOrCreate(appInstance, this::createAppStatusService, AppStatusService.class); + } + + @Override + public void close() throws IOException { + + } + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/AbstractStructureService.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/AbstractStructureService.java new file mode 100644 index 000000000..11844d0dd --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/AbstractStructureService.java @@ -0,0 +1,43 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure; + +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppSingletonIntegrationServiceImpl; + +public class AbstractStructureService extends AppSingletonIntegrationServiceImpl implements StructureService { + + private StructureIntegrationStandard appStandard; + + @Override + protected void initOperation(StructureOperation operation) { + operation.setStructureService(this); + operation.init(); + super.initOperation(operation); + } + + @Override + public void setAppStandard(StructureIntegrationStandard appStandard) { + this.appStandard = appStandard; + } + + @Override + public StructureIntegrationStandard getAppStandard() { + return appStandard; + } + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureIntegrationStandard.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureIntegrationStandard.java new file mode 100644 index 000000000..83e571326 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureIntegrationStandard.java @@ -0,0 +1,59 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure; + +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.app.structure.project.ProjectService; +import com.webank.wedatasphere.dss.standard.app.structure.role.RoleService; +import com.webank.wedatasphere.dss.standard.app.structure.status.AppStatusService; +import com.webank.wedatasphere.dss.standard.common.core.AppIntegrationStandard; +import com.webank.wedatasphere.dss.standard.common.core.AppStandard; +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; + + +public interface StructureIntegrationStandard extends AppIntegrationStandard { + + /** + * 统一角色规范，用于打通DSS与各集成接入系统的角色体系 + * @return + */ + RoleService getRoleService(AppInstance appInstance); + + /** + * 统一工程规范，用于打通DSS与各集成接入系统的工程体系 + * @return + */ + ProjectService getProjectService(AppInstance appInstance); + + AppStatusService getAppStateService(AppInstance appInstance); + + @Override + default String getStandardName() { + return "structureIntegrationStandard"; + } + + @Override + default int getGrade() { + return 2; + } + + @Override + default boolean isNecessary() { + return false; + } + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureOperation.java new file mode 100644 index 000000000..8b0fe5251 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureOperation.java @@ -0,0 +1,29 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure; + + +import com.webank.wedatasphere.dss.standard.common.service.Operation; + + +public interface StructureOperation extends Operation { + + void init(); + + void setStructureService(StructureService service); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureService.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureService.java new file mode 100644 index 000000000..f8c8a99fe --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/StructureService.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure; + + +import com.webank.wedatasphere.dss.standard.app.sso.request.SSORequestService; +import com.webank.wedatasphere.dss.standard.common.app.AppIntegrationService; + + +public interface StructureService extends AppIntegrationService { + + void setAppStandard(StructureIntegrationStandard appStandard); + + StructureIntegrationStandard getAppStandard(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectCreationOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectCreationOperation.java new file mode 100644 index 000000000..e75db6caa --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectCreationOperation.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + +public interface ProjectCreationOperation extends StructureOperation { + + ProjectResponseRef createProject(ProjectRequestRef projectRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectDeletionOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectDeletionOperation.java new file mode 100644 index 000000000..ef5f124f3 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectDeletionOperation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + +public interface ProjectDeletionOperation extends StructureOperation { + + ResponseRef deleteProject(ProjectRequestRef projectRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectRequestRef.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectRequestRef.java new file mode 100644 index 000000000..a0e991d9e --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectRequestRef.java @@ -0,0 +1,55 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.entity.ref.RequestRef; + + +public interface ProjectRequestRef extends RequestRef { + + String getWorkspaceName(); + + Long getId(); + + void setId(Long id); + + @Override + String getName(); + + void setName(String name); + + String getDescription(); + + void setDescription(String description); + + String getCreateBy(); + + void setCreateBy(String createBy); + + String getUpdateBy(); + + void setUpdateBy(String updateBy); + + default void setWorkspace(Workspace workspace){ + + } + + default Workspace getWorkspace(){ + return null; + } +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectRequestRefImpl.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectRequestRefImpl.java new file mode 100644 index 000000000..056b079e5 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectRequestRefImpl.java @@ -0,0 +1,110 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.sso.Workspace; +import com.webank.wedatasphere.dss.standard.common.entity.ref.AbstractRequestRef; + + +public class ProjectRequestRefImpl extends AbstractRequestRef implements ProjectRequestRef { + + + + private String workspaceName; + + private String createBy; + + private String updateBy; + + private Long id; + + private String name; + + private String description; + + private Workspace workspace; + + @Override + public String getWorkspaceName() { + return workspaceName; + } + + public void setWorkspaceName(String workspaceName) { + this.workspaceName = workspaceName; + } + + @Override + public Long getId() { + return this.id; + } + + @Override + public void setId(Long id) { + this.id = id; + } + + @Override + public String getName() { + return this.name; + } + + @Override + public void setName(String name) { + this.name = name; + } + + @Override + public String getDescription() { + return this.description; + } + + @Override + public void setDescription(String description) { + this.description = description; + } + + @Override + public String getCreateBy() { + return this.createBy; + } + + @Override + public void setCreateBy(String createBy) { + this.createBy = createBy; + } + + @Override + public String getUpdateBy() { + return updateBy; + } + + @Override + public void setUpdateBy(String updateBy) { + this.updateBy = updateBy; + } + + + @Override + public Workspace getWorkspace() { + return workspace; + } + + @Override + public void setWorkspace(Workspace workspace) { + this.workspace = workspace; + } +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectResponseRef.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectResponseRef.java new file mode 100644 index 000000000..efa91d53f --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectResponseRef.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +import java.util.List; +import java.util.Map; + + +public interface ProjectResponseRef extends ResponseRef { + + Long getProjectRefId(); + + Map getProjectRefIds(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectService.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectService.java new file mode 100644 index 000000000..ead407cb7 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectService.java @@ -0,0 +1,60 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.structure.AbstractStructureService; + + +public abstract class ProjectService extends AbstractStructureService { + + /** + * 是否支持DSS与各集成接入系统的协同开发能力 + * @return 默认为false + */ + public boolean isCooperationSupported() { + return false; + } + + public boolean isProjectNameUnique() { + return true; + } + + public ProjectCreationOperation getProjectCreationOperation() { + return getOrCreate(this::createProjectCreationOperation, ProjectCreationOperation.class); + } + + protected abstract ProjectCreationOperation createProjectCreationOperation(); + + public ProjectUpdateOperation getProjectUpdateOperation() { + return getOrCreate(this::createProjectUpdateOperation, ProjectUpdateOperation.class); + } + + protected abstract ProjectUpdateOperation createProjectUpdateOperation(); + + public ProjectDeletionOperation getProjectDeletionOperation() { + return getOrCreate(this::createProjectDeletionOperation, ProjectDeletionOperation.class); + } + + protected abstract ProjectDeletionOperation createProjectDeletionOperation(); + + public ProjectUrlOperation getProjectUrlOperation() { + return getOrCreate(this::createProjectUrlOperation, ProjectUrlOperation.class); + } + + protected abstract ProjectUrlOperation createProjectUrlOperation(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectUpdateOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectUpdateOperation.java new file mode 100644 index 000000000..a6f95bdcc --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectUpdateOperation.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + +public interface ProjectUpdateOperation extends StructureOperation { + + ProjectResponseRef updateProject(ProjectRequestRef projectRef) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectUrlOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectUrlOperation.java new file mode 100644 index 000000000..4d252ee6c --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/ProjectUrlOperation.java @@ -0,0 +1,27 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; +import com.webank.wedatasphere.dss.standard.common.exception.operation.ExternalOperationFailedException; + + +public interface ProjectUrlOperation extends StructureOperation { + + String getProjectUrl(ProjectRequestRef project) throws ExternalOperationFailedException; + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/Role.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/Role.java new file mode 100644 index 000000000..793cf57c8 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/Role.java @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import java.util.Date; + + +public interface Role { + + String getName(); + + String getShowName(); + + Date getCreateTime(); + + String getCreator(); + + String getDescription(); + + String getUpdator(); + + String getLastUpdateTime(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleCreationOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleCreationOperation.java new file mode 100644 index 000000000..f042e4916 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleCreationOperation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; + +import java.util.Map; + + +public interface RoleCreationOperation extends StructureOperation { + + RoleResponseRef createRole(String workspaceName, Role role); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleDeletionOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleDeletionOperation.java new file mode 100644 index 000000000..d3d6293de --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleDeletionOperation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; + +import java.util.Map; + + +public interface RoleDeletionOperation extends StructureOperation { + + RoleResponseRef deleteRole(String workspaceName, Role role); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleResponseRef.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleResponseRef.java new file mode 100644 index 000000000..b68da4b95 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleResponseRef.java @@ -0,0 +1,22 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +public interface RoleResponseRef extends ResponseRef { +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleService.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleService.java new file mode 100644 index 000000000..829441632 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleService.java @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureService; + + +public interface RoleService extends StructureService { + + RoleCreationOperation createRoleCreationOperation(); + + RoleUpdateOperation createRoleUpdateOperation(); + + RoleDeletionOperation createRoleDeletionOperation(); + + RoleUrlOperation createRoleUrlOperation(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleUpdateOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleUpdateOperation.java new file mode 100644 index 000000000..d33e53000 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleUpdateOperation.java @@ -0,0 +1,28 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; + +import java.util.Map; + + +public interface RoleUpdateOperation extends StructureOperation { + + RoleResponseRef updateRole(String workspaceName, Role fromRole, Role toRole); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleUrlOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleUrlOperation.java new file mode 100644 index 000000000..d8cb5ce8a --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/role/RoleUrlOperation.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.role; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; + + +public interface RoleUrlOperation extends StructureOperation { + + RoleResponseRef getRoleUrl(String workspaceName, Role role); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatus.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatus.java new file mode 100644 index 000000000..f583ce91e --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatus.java @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.status; + +import com.webank.wedatasphere.dss.standard.common.desc.AppInstance; + +import java.util.List; + + +public interface AppStatus { + + boolean isHealthy(); + + List getUnreachableAppInstance(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusOperation.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusOperation.java new file mode 100644 index 000000000..86168faab --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusOperation.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.status; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureOperation; + + +public interface AppStatusOperation extends StructureOperation { + + AppStatusResponseRef heartbeat(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusResponseRef.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusResponseRef.java new file mode 100644 index 000000000..bfe662b3c --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusResponseRef.java @@ -0,0 +1,25 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.status; + +import com.webank.wedatasphere.dss.standard.common.entity.ref.ResponseRef; + +public interface AppStatusResponseRef extends ResponseRef { + + AppStatus getAppStatus(); + +} diff --git a/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusService.java b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusService.java new file mode 100644 index 000000000..0448fea41 --- /dev/null +++ b/dss-standard/structure-standard/dss-structure-integration-standard/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/status/AppStatusService.java @@ -0,0 +1,26 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.status; + +import com.webank.wedatasphere.dss.standard.app.structure.StructureService; + + +public interface AppStatusService extends StructureService { + + AppStatusOperation createAppStatusOperation(); + +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/pom.xml b/dss-standard/structure-standard/spring-origin-dss-project-plugin/pom.xml new file mode 100644 index 000000000..f259abcd5 --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/pom.xml @@ -0,0 +1,84 @@ + + + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + ../../../pom.xml + + 4.0.0 + + spring-origin-dss-project-plugin + + + + com.webank.wedatasphere.dss + dss-project-plugin + ${dss.version} + + + org.springframework + spring-core + ${spring.version} + compile + + + org.springframework + spring-context + ${spring.version} + compile + + + org.springframework + spring-web + ${spring.version} + compile + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.8 + 1.8 + UTF-8 + + + + + + \ No newline at end of file diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectCooperationSpringFilter.java b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectCooperationSpringFilter.java new file mode 100644 index 000000000..447be5a61 --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/java/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/filter/ProjectCooperationSpringFilter.java @@ -0,0 +1,49 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.filter; + +import com.webank.wedatasphere.dss.standard.app.sso.plugin.filter.UserInterceptor; +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.ProjectCooperationPlugin; +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin.OriginProjectCooperationPlugin; +import javax.servlet.FilterConfig; +import org.springframework.stereotype.Component; +import org.springframework.web.context.WebApplicationContext; +import org.springframework.web.context.support.WebApplicationContextUtils; + + +@Component +public class ProjectCooperationSpringFilter extends ProjectCooperationFilter { + + @Override + public ProjectAuthInterceptor getProjectAuthInterceptor(FilterConfig filterConfig) { + WebApplicationContext webApplicationContext = + WebApplicationContextUtils.getRequiredWebApplicationContext(filterConfig.getServletContext()); + return webApplicationContext.getBean(ProjectAuthInterceptor.class); + } + + @Override + public UserInterceptor getUserAuthInterceptor(FilterConfig filterConfig) { + WebApplicationContext webApplicationContext = + WebApplicationContextUtils.getRequiredWebApplicationContext(filterConfig.getServletContext()); + return webApplicationContext.getBean(UserInterceptor.class); + } + + @Override + public ProjectCooperationPlugin getProjectCooperationPlugin(FilterConfig filterConfig) { + return OriginProjectCooperationPlugin.getProjectCooperatePlugin(); + } +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/OriginProjectCooperationPlugin.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/OriginProjectCooperationPlugin.scala new file mode 100644 index 000000000..5f9b0db61 --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/OriginProjectCooperationPlugin.scala @@ -0,0 +1,74 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard +import com.webank.wedatasphere.dss.standard.app.sso.builder.DssMsgBuilderOperation.DSSMsg +import com.webank.wedatasphere.dss.standard.app.sso.origin.OriginSSOIntegrationStandardFactory +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.{ProjectAuth, ProjectCooperationPlugin} +import com.webank.wedatasphere.linkis.httpclient.request.HttpAction +import javax.servlet.http.HttpServletRequest + + +class OriginProjectCooperationPlugin private() extends ProjectCooperationPlugin { + private val ssoIntegrationStandard:SSOIntegrationStandard =new OriginSSOIntegrationStandardFactory().getSSOIntegrationStandard + + private def getProjectAuth(action: HttpAction, dssMsg: DSSMsg): ProjectAuthImpl = { + val dwsHttpClient = HttpClient.getDSSClient(dssMsg.getDSSUrl) + val projectAuth = new ProjectAuthImpl + projectAuth.setWorkspaceName(dssMsg.getWorkspaceName) + HttpClient.addCookies(dssMsg, action) + dwsHttpClient.execute(action) match { + case projectAuthResult: ProjectAuthResult => + projectAuth.setProjectId(projectAuthResult.getProjectId) + projectAuth.setProjectName(projectAuthResult.getProjectName) + projectAuth.setEditUsers(projectAuthResult.getEditUsers) + projectAuth.setAccessUsers(projectAuthResult.getAccessUsers) + projectAuth.setDeleteUsers(projectAuthResult.getDeleteUsers) + } + projectAuth + } + + override def getProjectAuth(request: HttpServletRequest, projectId: String): ProjectAuth = { + val dssMsg = ssoIntegrationStandard.getSSOPluginService + .createDssMsgCacheOperation().getDSSMsgInSession(request) + val projectAuthAction = new ProjectAuthByIdAction + projectAuthAction.setWorkspace(dssMsg.getWorkspaceName) + projectAuthAction.setProjectId(projectId) + projectAuthAction.setComponentName(dssMsg.getAppName) + val projectAuth = getProjectAuth(projectAuthAction, dssMsg) + projectAuth.setProjectId(projectId) + projectAuth + } + + override def getProjectAuthByName(request: HttpServletRequest, projectName: String): ProjectAuth = { + + val dssMsg = ssoIntegrationStandard.getSSOPluginService + .createDssMsgCacheOperation().getDSSMsgInSession(request) + val projectAuthAction = new ProjectAuthByNameAction + projectAuthAction.setProjectName(projectName) + projectAuthAction.setWorkspace(dssMsg.getWorkspaceName) + val projectAuth = getProjectAuth(projectAuthAction, dssMsg) + projectAuth.setProjectName(projectName) + projectAuth + } +} +object OriginProjectCooperationPlugin { + private val projectCooperatePlugin = new OriginProjectCooperationPlugin + def getProjectCooperatePlugin: ProjectCooperationPlugin = projectCooperatePlugin +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/OriginProjectPlugin.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/OriginProjectPlugin.scala new file mode 100644 index 000000000..8e23ada1d --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/OriginProjectPlugin.scala @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import java.util + +import com.webank.wedatasphere.dss.standard.app.sso.SSOIntegrationStandard +import com.webank.wedatasphere.dss.standard.app.sso.origin.OriginSSOIntegrationStandardFactory +import com.webank.wedatasphere.dss.standard.app.sso.origin.client.HttpClient +import com.webank.wedatasphere.dss.standard.app.structure.project.plugin.ProjectPlugin +import javax.servlet.http.HttpServletRequest + + +class OriginProjectPlugin private() extends ProjectPlugin { + + override def getProjects(request: HttpServletRequest): util.List[String] = { + val ssoIntegrationStandard:SSOIntegrationStandard =new OriginSSOIntegrationStandardFactory().getSSOIntegrationStandard + val dssMsg =ssoIntegrationStandard.getSSOPluginService.createDssMsgCacheOperation().getDSSMsgInSession(request) + val dwsHttpClient = HttpClient.getDSSClient(dssMsg.getDSSUrl) + val projectListAction = new ProjectListByWorkspaceAction + projectListAction.setWorkspace(dssMsg.getWorkspaceName) + HttpClient.addCookies(dssMsg, projectListAction) + dwsHttpClient.execute(projectListAction) match { + case projectListResult: ProjectListByWorkspaceResult => + projectListResult.getProjectIds + } + } + +} +object OriginProjectPlugin { + private val projectPlugin = new OriginProjectPlugin + def getProjectPlugin: ProjectPlugin = projectPlugin +} \ No newline at end of file diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthByIdAction.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthByIdAction.scala new file mode 100644 index 000000000..11ee6bb0d --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthByIdAction.scala @@ -0,0 +1,34 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction + + +class ProjectAuthByIdAction extends POSTAction with DWSHttpAction { + + override def suffixURLs: Array[String] = Array("dss", "getProjectAuthOfWorkspace") + + def setWorkspace(workspaceName: String): Unit = addRequestPayload("workspaceName", workspaceName) + + def setProjectId(projectId: String): Unit = addRequestPayload("projectId", projectId) + + def setComponentName(componentName: String): Unit = addRequestPayload("componentName", componentName) + + override def getRequestPayload: String = "" +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthByNameAction.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthByNameAction.scala new file mode 100644 index 000000000..22428aadc --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthByNameAction.scala @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction + + +class ProjectAuthByNameAction extends POSTAction with DWSHttpAction { + + override def suffixURLs: Array[String] = Array("dss", "getProjectAuthOfWorkspace") + + def setWorkspace(workspaceName: String): Unit = addRequestPayload("workspaceName", workspaceName) + + def setProjectName(projectName: String): Unit = addRequestPayload("projectName", projectName) + + override def getRequestPayload: String = "" +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthResult.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthResult.scala new file mode 100644 index 000000000..79d810479 --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectAuthResult.scala @@ -0,0 +1,47 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import scala.beans.BeanProperty +import java.util + +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/getProjectAuthOfWorkspace") +class ProjectAuthResult extends DWSResult { + + @BeanProperty + var projectName: String = _ + + @BeanProperty + var projectId: String = _ + + @BeanProperty + var workspaceName: String = _ + + @BeanProperty + var editUsers: util.List[String] = _ + + @BeanProperty + var accessUsers: util.List[String] = _ + + @BeanProperty + var deleteUsers: util.List[String] = _ + +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectListByWorkspaceAction.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectListByWorkspaceAction.scala new file mode 100644 index 000000000..34bb6e031 --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectListByWorkspaceAction.scala @@ -0,0 +1,30 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import com.webank.wedatasphere.linkis.httpclient.dws.request.DWSHttpAction +import com.webank.wedatasphere.linkis.httpclient.request.POSTAction + + +class ProjectListByWorkspaceAction extends POSTAction with DWSHttpAction { + + override def suffixURLs: Array[String] = Array("dss", "getProjectListOfWorkspace") + + def setWorkspace(workspaceName: String): Unit = addRequestPayload("workspaceName", workspaceName) + + override def getRequestPayload: String = "" +} diff --git a/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectListByWorkspaceResult.scala b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectListByWorkspaceResult.scala new file mode 100644 index 000000000..8114655b1 --- /dev/null +++ b/dss-standard/structure-standard/spring-origin-dss-project-plugin/src/main/scala/com/webank/wedatasphere/dss/standard/app/structure/project/plugin/origin/ProjectListByWorkspaceResult.scala @@ -0,0 +1,33 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.dss.standard.app.structure.project.plugin.origin + +import java.util + +import com.webank.wedatasphere.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import com.webank.wedatasphere.linkis.httpclient.dws.response.DWSResult + +import scala.beans.BeanProperty + + +@DWSHttpMessageResult("/api/rest_j/v\\d+/dss/getProjectListOfWorkspace") +class ProjectListByWorkspaceResult extends DWSResult { + + @BeanProperty + var projectIds: util.List[String] = _ + +} diff --git a/dss-user-manager/pom.xml b/dss-user-manager/pom.xml deleted file mode 100644 index ef03b3a66..000000000 --- a/dss-user-manager/pom.xml +++ /dev/null @@ -1,87 +0,0 @@ - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-user-manager - jar - - UTF-8 - - - - - com.webank.wedatasphere.linkis - linkis-module - ${linkis.version} - provided - - - scala-xml_2.11 - org.scala-lang.modules - - - guava - com.google.guava - - - - - com.webank.wedatasphere.linkis - linkis-common - 0.9.4 - - - com.github.rholder - guava-retrying - 2.0.0 - provided - - - org.junit.jupiter - junit-jupiter - RELEASE - test - - - dom4j - dom4j - 1.6.1 - - - cn.hutool - hutool-all - 5.3.2 - - - mysql - mysql-connector-java - 5.1.49 - - - com.typesafe - config - 1.4.1 - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - org.apache.maven.plugins - maven-jar-plugin - - - - - \ No newline at end of file diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/conf/DSSUserManagerConfig.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/conf/DSSUserManagerConfig.java deleted file mode 100644 index 731194cc4..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/conf/DSSUserManagerConfig.java +++ /dev/null @@ -1,60 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.conf; - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; - -import java.util.ResourceBundle; - -/** - * @program: dss-appjoint-auth - * @description: 用户模块配置文件 - * - * @create: 2020-12-30 16:26 - **/ - - -public class DSSUserManagerConfig { -// private final static ResourceBundle resource = ResourceBundle.getBundle("linkis"); - public static final String LOCAL_USER_ROOT_PATH = CommonVars.apply("wds.dss.user.root.dir","null").getValue().trim(); - public static final String BDP_SERVER_MYBATIS_DATASOURCE_URL = CommonVars.apply("wds.linkis.server.mybatis.datasource.url", "null").getValue().trim(); - public static final String BDP_SERVER_MYBATIS_DATASOURCE_USERNAME = CommonVars.apply("wds.linkis.server.mybatis.datasource.username", "null").getValue().trim(); - public static final String BDP_SERVER_MYBATIS_DATASOURCE_PASSWORD = CommonVars.apply("wds.linkis.server.mybatis.datasource.password", "null").getValue().trim(); - public static final String SCHEDULER_ADDRESS = CommonVars.apply("wds.dss.appjoint.scheduler.azkaban.address", "null").getValue().trim(); - public static final String USER_ACCOUNT_COMMANDS = CommonVars.apply("wds.dss.user.account.command.class", "null").getValue().trim(); - - public static final String METASTORE_HDFS_PATH = CommonVars.apply("wds.linkis.metastore.hive.hdfs.base.path", "null").getValue().trim(); - public static final String METASTORE_SCRIPT_PAHT = CommonVars.apply("wds.linkis.metastore.script.path", "null").getValue().trim(); - public static final String METASTORE_DB_TAIL = CommonVars.apply("wds.linkis.metastore.db.tail", "_default").getValue().trim(); - - public static final String KERBEROS_REALM = CommonVars.apply("wds.linkis.kerberos.realm", "null").getValue().trim(); - public static final String KERBEROS_ADMIN = CommonVars.apply("wds.linkis.kerberos.admin", "null").getValue().trim(); - public static final String KERBEROS_SCRIPT_PATH = CommonVars.apply("wds.linkis.kerberos.script.path", "null").getValue().trim(); - public static final String KERBEROS_KEYTAB_PATH = CommonVars.apply("wds.linkis.kerberos.keytab.path", "null").getValue().trim(); - public static final String KERBEROS_SSH_PORT = CommonVars.apply("wds.linkis.kerberos.ssh.port", "22").getValue().trim(); - public static final String KERBEROS_KDC_NODE = CommonVars.apply("wds.linkis.kerberos.kdc.node", "null").getValue().trim(); - public static final String KERBEROS_KDC_USER_NAME = CommonVars.apply("wds.linkis.kerberos.kdc.user.name", "null").getValue().trim(); - public static final String KERBEROS_KDC_USER_PASSWORD = CommonVars.apply("wds.linkis.kerberos.kdc.user.password", "null").getValue().trim(); - public static final String KERBEROS_ENABLE_SWITCH = CommonVars.apply("wds.linkis.kerberos.enable.switch", "null").getValue().trim(); - public static final String DSS_DEPLOY_PATH = CommonVars.apply("wds.dss.deploy.path", "null").getValue().trim(); - public static final String DSS_SCHEDULER_URL = CommonVars.apply("wds.dss.scheduler.url", "/schedule/system").getValue().trim(); - - - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/dto/request/AuthorizationBody.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/dto/request/AuthorizationBody.java deleted file mode 100644 index 250f66f16..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/dto/request/AuthorizationBody.java +++ /dev/null @@ -1,99 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.dto.request; - -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; - -/** - * @program: user-manager - * @description: 施工单数据结构 - * - * @create: 2020-08-12 14:29 - **/ -public class AuthorizationBody { - - private String username; - private String password; - private String dssInstallDir; - private String azkakanDir; - private ArrayList servers; - - - public ArrayList getServers() { - return servers; - } - - public void setServers(ArrayList servers) { - this.servers = servers; - } - - - - - public String getAzkakanDir() { - return azkakanDir; - } - - public void setAzkakanDir(String azkakanDir) { - this.azkakanDir = azkakanDir; - } - - public String getDssInstallDir() { - return dssInstallDir; - } - - public void setDssInstallDir(String installDir) { - this.dssInstallDir = installDir; - } - - public List> getPaths() { - return paths; - } - - public void setPaths(List> paths) { - this.paths = paths; - } - - private List> paths; - - public String getUsername() { - return username; - } - - public void setUsername(String username) { - this.username = username; - } - - public String getPassword() { - return password; - } - - public void setPassword(String password) { - this.password = password; - } - - public String getDatabaseName(){ - return this.username + "_default"; - } - - - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/dto/request/LinuxServer.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/dto/request/LinuxServer.java deleted file mode 100644 index 7569612f5..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/dto/request/LinuxServer.java +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.dto.request; - -public class LinuxServer { - - private String linuxHost; //master ip，Comma separated - private String linuxLoginUser; - private String linuxLoginPassword; - - - public String getLinuxHost() { - return linuxHost; - } - - public void setLinuxHost(String linuxHosts) { - this.linuxHost = linuxHosts; - } - - public String getLinuxLoginUser() { - return linuxLoginUser; - } - - public void setLinuxLoginUser(String linuxLoginUser) { - this.linuxLoginUser = linuxLoginUser; - } - - public String getLinuxLoginPassword() { - return linuxLoginPassword; - } - - public void setLinuxLoginPassword(String linuxLoginPassword) { - this.linuxLoginPassword = linuxLoginPassword; - } - - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/AbsCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/AbsCommand.java deleted file mode 100644 index be4bf3332..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/AbsCommand.java +++ /dev/null @@ -1,114 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service; - - -import com.webank.wedatasphere.linkis.server.Message; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import org.dom4j.DocumentException; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import javax.ws.rs.core.Response; -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStreamReader; -import java.net.URL; - -/** - * 各模块的授权 继承这个类 根据需要实现自己的类。 - */ -public abstract class AbsCommand implements Command { - protected final Logger logger = LoggerFactory.getLogger(getClass()); - - @Override - public String capacity(AuthorizationBody body) { - return Command.SUCCESS; - } - - @Override - public String renew(AuthorizationBody body) { - return Command.SUCCESS; - } - - @Override - public String undoAuthorization(AuthorizationBody body) { return Command.SUCCESS; } - - @Override -// public String authorization(AuthorizationBody body) throws DocumentException { return Command.SUCCESS; } - public String authorization(AuthorizationBody body) throws IOException, Exception { return Command.SUCCESS; } - - public String toMessage(String msg) { - return this.getClass().getSimpleName() + "the module starts execution"+ msg; - } - - protected String runShell(String scriptPath, String[] args){ - String bashCommand; - try { - bashCommand = "sh " + scriptPath + " " + String.join(" ", args); - Runtime runtime = Runtime.getRuntime(); - Process process = runtime.exec(bashCommand); - - return this.getString(process); - } - catch (Exception e){ - logger.error(scriptPath, e); - return e.getMessage(); - } - } - - protected String getString(Process process) throws IOException, InterruptedException { - BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream())); - - String inline; - while ((inline = br.readLine()) != null) { - if (!inline.equals("")) { - inline = inline.replaceAll("<", "<").replaceAll(">", ">"); - logger.info(inline); - } else { - logger.info("\n"); - } - } - br.close(); - br = new BufferedReader(new InputStreamReader(process.getErrorStream())); //错误信息 - while ((inline = br.readLine()) != null) { - if (!inline.equals("")) - logger.warn(inline); - else - logger.warn("\n"); - } - - int status = process.waitFor(); - if (status != 0){ - logger.error("shell error: "+status); - } - br.close(); - return Command.SUCCESS; - } - - protected String getResource(String path){ - try { - URL url = this.getClass().getClassLoader().getResource(path); - return url.getPath(); - }catch (Exception e){ - logger.error("File does not exist " + path, e); - } - return null; - } -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/Command.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/Command.java deleted file mode 100644 index dfed25446..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/Command.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service; - - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import org.dom4j.DocumentException; - -import java.io.IOException; - -public interface Command { - - final public static String SUCCESS = "success"; - final public static String ERROR = "error"; - /** - * 授权开通服务 - * @param body - * @return 成功 success 其他失败 - */ - public String authorization(AuthorizationBody body) throws DocumentException, IOException, Exception; - - /** - * 关闭授权 - * @param body - * @return 成功 success 其他失败 - */ - public String undoAuthorization(AuthorizationBody body) throws Exception; - - /** - * 扩容 - * @param body - * @return 成功 success 其他失败 - */ - public String capacity(AuthorizationBody body) throws Exception; - - /** - * 续费 - * @param body - * @return 成功 success 其他失败 - */ - public String renew(AuthorizationBody body) throws Exception; -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/MacroCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/MacroCommand.java deleted file mode 100644 index 0d2d3431c..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/MacroCommand.java +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service; - - -/** - * @program: user-authorization - * @description: 开通命令接口 - * - * @create: 2020-08-10 14:24 - **/ -public interface MacroCommand extends Command { - - public void add(AbsCommand command) throws Exception; - - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/AzkabanCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/AzkabanCommand.java deleted file mode 100644 index 2f85ae38b..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/AzkabanCommand.java +++ /dev/null @@ -1,110 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import org.dom4j.Document; -import org.dom4j.DocumentException; -import org.dom4j.Element; -import org.dom4j.io.OutputFormat; -import org.dom4j.io.SAXReader; -import org.dom4j.io.XMLWriter; - -import java.io.File; -import java.io.FileInputStream; -import java.io.FileOutputStream; -import java.io.IOException; -import java.util.Iterator; -import java.util.List; - -/** - * @program: dss-appjoint-auth - * @description: 开通azkaban账号 - * - * @create: 2021-01-08 15:53 - **/ - -public class AzkabanCommand extends AbsCommand { - @Override - public String authorization(AuthorizationBody body) { - - try{ - this.xmlHandler(body.getAzkakanDir()+"/conf/azkaban-users.xml", body); - String[] args = {body.getUsername(), body.getPassword(), body.getDssInstallDir()+"/conf/"}; - String path = getResource("default/AddschedulerUser.sh"); - return this.runShell(path, args); - }catch (Exception err){ - logger.error("AzkabanCommand auth error:", err); - return err.getMessage(); - } - } - - private void xmlHandler(String azkPath, AuthorizationBody body) throws DocumentException, IOException { - SAXReader reader = new SAXReader(); - - File file = new File(azkPath); - Document document; - FileInputStream fis = null; - try { - fis = new FileInputStream(file); - document = reader.read(fis); - }catch (DocumentException e){ - throw e; - }finally { - if (fis != null) { - fis.close(); - } - } - - Element root = document.getRootElement(); - - Iterator it = root.elementIterator("user"); - Boolean userExists = false; - Element element = null; - while (it.hasNext()) { - element = (Element) it.next(); - - String v = element.attributeValue("username"); - if(v.equals(body.getUsername())){ //修改密码 - userExists = true; - element.attribute("password").setValue(body.getPassword()); - } - } - if(!userExists){ //新增账号 - Element cloneEl = element.createCopy(); - cloneEl.attribute("username").setValue(body.getUsername()); - cloneEl.attribute("password").setValue(body.getPassword()); - - List elements = root.elements("user"); - elements.add(elements.size(), cloneEl); - } - - this.saveXml(document, file); - - } - - private void saveXml(Document document, File file) throws IOException { - FileOutputStream out =new FileOutputStream(file); - OutputFormat format=OutputFormat.createPrettyPrint(); - XMLWriter writer=new XMLWriter(out, format); - writer.write(document); - writer.close(); - } -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/KerberosCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/KerberosCommand.java deleted file mode 100644 index ef120f19e..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/KerberosCommand.java +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.typesafe.config.Config; -import com.typesafe.config.ConfigFactory; -import com.webank.wedatasphere.linkis.common.conf.CommonVars; -import com.webank.wedatasphpere.dss.user.conf.DSSUserManagerConfig; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; - -import java.io.IOException; -import java.net.InetAddress; - -/** - * @date 2021/1/5 - */ -public class KerberosCommand extends AbsCommand { - - @Override - public String authorization(AuthorizationBody body) throws Exception { - String rst = createKt(body); - return rst != Command.SUCCESS ? rst : Command.SUCCESS; - } - - private String createKt(AuthorizationBody body) throws Exception { - String userName = body.getUsername(); - String hostName = InetAddress.getLocalHost().getHostName(); - String res = null; - if(userName != null){ - res = callShell(DSSUserManagerConfig.KERBEROS_SCRIPT_PATH, userName,hostName, - DSSUserManagerConfig.KERBEROS_KEYTAB_PATH,DSSUserManagerConfig.KERBEROS_SSH_PORT, - DSSUserManagerConfig.KERBEROS_KDC_NODE,DSSUserManagerConfig.KERBEROS_KDC_USER_NAME,DSSUserManagerConfig.KERBEROS_KDC_USER_PASSWORD,DSSUserManagerConfig.KERBEROS_REALM,DSSUserManagerConfig.KERBEROS_ENABLE_SWITCH); - } - return res; - } - - private String callShell(String shellFile, String username, String hostName, String keytabPath, - String sshPort, String kdcNode, String kdcUser,String password, String realm,String enableSwich) throws Exception { - - String bashCommand = getResource(shellFile); - String scriptCmd ; - if(null != hostName){ - scriptCmd = String.format("%s %s %s %s %s %s %s %s %s", username,hostName,keytabPath,sshPort,kdcNode,kdcUser,password,realm,enableSwich); - }else { - scriptCmd = String.format("%s %s %s %s %s %s %s %s", username,keytabPath,sshPort,kdcNode,kdcUser,password,realm,enableSwich); - } - String[] args = scriptCmd.split(" "); - return this.runShell(bashCommand, args); - } - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/LdapCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/LdapCommand.java deleted file mode 100644 index 31c2ecdbb..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/LdapCommand.java +++ /dev/null @@ -1,58 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; -import com.webank.wedatasphpere.dss.user.conf.DSSUserManagerConfig; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; - -import java.io.BufferedReader; -import java.io.InputStreamReader; -import java.util.HashMap; -import java.util.List; - -/** - * @program: user-authorization - * @description: 创建用户空间 - * - * @create: 2020-08-13 13:39 - **/ - -public class LdapCommand extends AbsCommand { - - @Override - public String authorization(AuthorizationBody body) throws Exception { - - String userName = body.getUsername(); - String UserPassword = body.getPassword(); - String dssDeployPath = DSSUserManagerConfig.DSS_DEPLOY_PATH; - - String bashCommand = this.getClass().getClassLoader().getResource("default/CreateLdapAccount.sh").getPath(); - String[] args = { - userName, - UserPassword, - dssDeployPath - }; - - return this.runShell(bashCommand, args); - } - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/LinuxUserCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/LinuxUserCommand.java deleted file mode 100644 index 10a567e21..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/LinuxUserCommand.java +++ /dev/null @@ -1,60 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.dto.request.LinuxServer; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; - -import java.util.ArrayList; - -/** - * @program: user-authorization - * @description: 创建用户空间 - * - * @create: 2020-08-13 13:39 - **/ - -public class LinuxUserCommand extends AbsCommand { - - @Override - public String authorization(AuthorizationBody body) throws Exception { - -// String hosts = body.getLinuxHosts(); -// String linuxPassword = body.getLinuxLoginPassword(); -// String linuxUserName = body.getLinuxLoginUser(); - ArrayList linuxServers = body.getServers(); - logger.info("服务器ip"+linuxServers.toString()); - StringBuffer stringBuffer = new StringBuffer(); - for(LinuxServer linuxServer:linuxServers){ - String hosts = linuxServer.getLinuxHost(); - String linuxPassword = linuxServer.getLinuxLoginPassword(); - String linuxUserName = linuxServer.getLinuxLoginUser(); - stringBuffer.append(hosts).append("#").append(linuxUserName).append("#").append(linuxPassword).append(","); - } - stringBuffer.deleteCharAt(stringBuffer.lastIndexOf(",")); - String addUserName = body.getUsername(); - String addUserPassword = body.getPassword(); - String bashCommand = this.getClass().getClassLoader().getResource("default/CreateLinuxUser.sh").getPath(); - String[] args = {stringBuffer.toString(),addUserName,addUserPassword}; - - return this.runShell(bashCommand, args); - } - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/MetastoreCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/MetastoreCommand.java deleted file mode 100644 index 607691919..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/MetastoreCommand.java +++ /dev/null @@ -1,56 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.typesafe.config.Config; -import com.typesafe.config.ConfigFactory; -import com.webank.wedatasphere.linkis.common.conf.CommonVars; -import com.webank.wedatasphpere.dss.user.conf.DSSUserManagerConfig; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * @date 2021/1/5 - */ -public class MetastoreCommand extends AbsCommand { - private static final Logger logger = LoggerFactory.getLogger(MetastoreCommand.class); - - @Override - public String authorization(AuthorizationBody body) throws Exception { - String rst = createDb(body); - return rst != Command.SUCCESS ? rst : Command.SUCCESS; - } - - private String createDb(AuthorizationBody body) throws Exception { - String bashCommand = null; - String[] args = null; - String userName = body.getUsername(); - if (userName != null) { - String dbName = userName + DSSUserManagerConfig.METASTORE_DB_TAIL; - String path = DSSUserManagerConfig.METASTORE_HDFS_PATH + "/"+dbName+".db"; - bashCommand = getResource(DSSUserManagerConfig.METASTORE_SCRIPT_PAHT); - args = new String[]{ userName,dbName,path, - DSSUserManagerConfig.KERBEROS_REALM,DSSUserManagerConfig.KERBEROS_ADMIN,DSSUserManagerConfig.KERBEROS_KEYTAB_PATH,DSSUserManagerConfig.KERBEROS_ENABLE_SWITCH}; - } - return this.runShell(bashCommand, args); - } -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/SchedulisCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/SchedulisCommand.java deleted file mode 100644 index 73351943b..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/SchedulisCommand.java +++ /dev/null @@ -1,146 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.fasterxml.jackson.databind.JsonNode; -import com.webank.wedatasphpere.dss.user.conf.DSSUserManagerConfig; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; - -import java.net.URI; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.PreparedStatement; -import java.sql.SQLException; -import java.sql.Statement; -import java.util.Arrays; -import java.util.HashMap; -import java.util.Map; - -import cn.hutool.crypto.SecureUtil; -import cn.hutool.crypto.symmetric.DES; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.http.HttpStatus; -import org.springframework.http.ResponseEntity; -import org.springframework.util.StringUtils; -import org.springframework.web.client.RestTemplate; -import org.springframework.web.util.UriComponentsBuilder; - -/** - * @date 2021/1/11 - */ -public class SchedulisCommand extends AbsCommand { - - private static final Logger logger = LoggerFactory.getLogger(SchedulisCommand.class); - - @Override - public String authorization(AuthorizationBody body) { - String username = body.getUsername(); - logger.info("开始新增调度用户：{}", username); - String password = body.getPassword(); - String encryptionPwd = getEncryptionPwd(username, password); - Connection connection; - Statement stmt; - try { - logger.info("开始插入ctyun_user表"); - connection = getConnection(); - stmt = connection.createStatement(); - String sql = "INSERT INTO `ctyun_user` (`id`,`name`,`username`,`email`,`password`,`work_order_item_config`) VALUES (?,?,?,?,?,NULL) ON DUPLICATE KEY UPDATE `password` = ?"; - PreparedStatement statement = connection.prepareCall(sql); - statement.setString(1, username); - statement.setString(2, username); - statement.setString(3, username); - statement.setString(4, username); - statement.setString(5, encryptionPwd); - statement.setString(6, encryptionPwd); - statement.executeUpdate(); - stmt.close(); - connection.close(); - logger.info("完成插入ctyun_user表"); - } catch (SQLException e) { - logger.error(e.getMessage()); - } - logger.info("开始调用接口新增schedulis用户"); - addSchedulisUser(username, password); - logger.info("结束调用接口新增schedulis用户"); - return Command.SUCCESS; - } - - private static Connection getConnection() { - try { - //注册数据库的驱动 - Class.forName("com.mysql.jdbc.Driver"); - //获取数据库连接（里面内容依次是：主机名和端口、用户名、密码） - String url = DSSUserManagerConfig.BDP_SERVER_MYBATIS_DATASOURCE_URL; - String user = DSSUserManagerConfig.BDP_SERVER_MYBATIS_DATASOURCE_USERNAME; - String password = DSSUserManagerConfig.BDP_SERVER_MYBATIS_DATASOURCE_PASSWORD; - return DriverManager.getConnection(url, user, password); - } catch (Exception e) { - logger.error(e.getMessage()); - } - return null; - } - - private boolean addSchedulisUser(String username, String password) { - String schedulerUrl = DSSUserManagerConfig.DSS_SCHEDULER_URL; - RestTemplate restTemplate = new RestTemplate(); - String schedulisUrl = DSSUserManagerConfig.SCHEDULER_ADDRESS; - String url = new StringBuilder().append(schedulisUrl) - .append(schedulerUrl) - .toString(); - Map params = new HashMap<>(4); - params.put("userId", username); - params.put("password", password); - params.put("ajax", "addSystemUserViaFastTrackCtyun"); - String fullUrl = addParams(url, params); - UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(fullUrl); - URI uri = builder.build().encode().toUri(); - ResponseEntity responseEntity = restTemplate.getForEntity(uri, JsonNode.class); - return HttpStatus.OK.equals(responseEntity.getStatusCode()); - } - - private static String addParams(String url, Map params) { - if (params.isEmpty()) { - return url; - } - StringBuilder sb = new StringBuilder().append(url).append("?"); - for (Map.Entry entry : params.entrySet()) { - if (StringUtils.hasText(entry.getValue())) { - sb.append(entry.getKey()) - .append("=") - .append(entry.getValue()) - .append("&"); - } - } - return sb.deleteCharAt(sb.length()-1).toString(); - } - - private String getEncryptionPwd(String username, String password) { - int minSize = 8; - while (username.length() < minSize) { - username += username; - } - byte[] keyBytes = username.getBytes(); - Arrays.sort(keyBytes); - DES des = SecureUtil.des(keyBytes); - return des.encryptHex(password); - } -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/UserAuthorizationClient.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/UserAuthorizationClient.java deleted file mode 100644 index fd275a0e4..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/UserAuthorizationClient.java +++ /dev/null @@ -1,57 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - - -import com.webank.wedatasphpere.dss.user.conf.DSSUserManagerConfig; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * @program: user-authorization - * @description: - * - * @create: 2020-08-10 14:24 - **/ -public class UserAuthorizationClient { - - public UserMacroCommand userMacroCommand = new UserMacroCommand(); - protected final Logger logger = LoggerFactory.getLogger(UserAuthorizationClient.class); - - public UserAuthorizationClient() { - - String[] commandPaths = DSSUserManagerConfig.USER_ACCOUNT_COMMANDS.split(","); - for(String classPath: commandPaths){ - try { - userMacroCommand.add((AbsCommand) Class.forName(classPath).newInstance()); - } catch (Exception e) { - logger.info(e.getMessage()); - e.printStackTrace(); - } - } - } - - public String authorization(AuthorizationBody body) throws Exception { - return userMacroCommand.authorization(body); - } - - -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/UserMacroCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/UserMacroCommand.java deleted file mode 100644 index c842e666c..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/UserMacroCommand.java +++ /dev/null @@ -1,118 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - - -import com.github.rholder.retry.*; -import com.google.common.base.Predicates; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; -import com.webank.wedatasphpere.dss.user.service.MacroCommand; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.IOException; -import java.util.ArrayList; -import java.util.List; -import java.util.concurrent.Callable; -import java.util.concurrent.ExecutionException; - -/** - * @program: user-authorization - * @description: 开通命令实现 - * - * @create: 2020-08-10 14:24 - **/ -public class UserMacroCommand implements MacroCommand { - - private List commandList = new ArrayList<>(); - - protected final Logger logger = LoggerFactory.getLogger(getClass()); - - private static Integer RETRY_COUNT = 1; - - @Override - public void add(AbsCommand command) { - - commandList.add(command); - } - - @Override - public String authorization(AuthorizationBody body) throws Exception { - - return this.execute("authorization", body); - - } - - @Override - public String undoAuthorization(AuthorizationBody json) throws Exception { - - return this.execute("undoAuthorization", json); - - } - - @Override - public String capacity(AuthorizationBody json) throws Exception { - - return this.execute("capacity", json); - - } - - @Override - public String renew(AuthorizationBody json) throws Exception { - - return this.execute("renew", json); - - } - - - /** - * 授权操作基础方法 - * @param funName 调用的函数名 - * @param body 传入的数据 - * @return - * @throws ClassNotFoundException - * @throws NoSuchMethodException - */ - private String execute(String funName, AuthorizationBody body) throws Exception { - - for (AbsCommand command : commandList) { - - switch (funName) { - case "authorization": - command.authorization(body); - - case "undoAuthorization": - command.undoAuthorization(body); - - case "capacity": - command.capacity(body); - } - - } - - return "success"; - } - - - } - - - diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/WorkspaceCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/WorkspaceCommand.java deleted file mode 100644 index 37b9f9597..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/WorkspaceCommand.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; - -import java.util.HashMap; -import java.util.List; - -/** - * @program: user-authorization - * @description: 创建用户空间 - * - * @create: 2020-08-13 13:39 - **/ - -public class WorkspaceCommand extends AbsCommand { - - @Override - public String authorization(AuthorizationBody body) throws Exception { - List> paths = body.getPaths(); - String result = ""; - Boolean isSuccess = true; - for(HashMap map : paths){ - String path = map.get("value"); - String rst = createDir(path, body); - result += rst; - if(!rst.equals(Command.SUCCESS)){ - isSuccess = false; - } - } - if(isSuccess){ - return Command.SUCCESS; - } - logger.error(result); - return result; - } - - private String createDir(String path, AuthorizationBody body) throws Exception { - String bashCommand; - - if(path.contains("hdfs:")){ - path = path.replace("hdfs://", "") + "/" + body.getUsername(); - bashCommand = getResource("default/HdfsPath.sh"); - }else { - path = path.replace("file://", "") + "/" + body.getUsername(); - bashCommand = getResource("default/LinuxPath.sh"); - } - String[] args = {body.getUsername(), path}; - return this.runShell(bashCommand, args); - } -} diff --git a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/YarnCommand.java b/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/YarnCommand.java deleted file mode 100644 index a4d854780..000000000 --- a/dss-user-manager/src/main/java/com/webank/wedatasphpere/dss/user/service/impl/YarnCommand.java +++ /dev/null @@ -1,39 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import com.webank.wedatasphpere.dss.user.service.AbsCommand; -import com.webank.wedatasphpere.dss.user.service.Command; - -/** - * @program: user-authorization - * @description: 开通yarn权限的实现 - * - * @create: 2020-08-10 14:24 - **/ -public class YarnCommand extends AbsCommand { - - @Override - public String authorization(AuthorizationBody body) { - - return Command.SUCCESS; - } -} diff --git a/dss-user-manager/src/main/resources/config/properties.conf b/dss-user-manager/src/main/resources/config/properties.conf deleted file mode 100644 index ece97f39a..000000000 --- a/dss-user-manager/src/main/resources/config/properties.conf +++ /dev/null @@ -1,14 +0,0 @@ -#--Metastore配置 -base_path = #--元数据位于hdfs位置上的base -metastore_sh = default/Metastore.sh #--执行脚本的位置 -db_tail = #--hdfs上数据库文件的尾缀 -realm = #--kerberos域 -admin = #--获得hive metastore服务admin角色授权的用户（如hdfs,非常重要，否则无法完成授权） - -#--Kerberos配置 -shellFile = default/Kerberos.sh #--执行脚本的位置 -keytabPath = /etc/security/keytabs #--keytab在本节点上的存放位置 -sshPort = 22 #--ssh操作使用的端口 -kdcNode = #--kerberos服务kdc节点 -kdcUser = #--在进行ssh kdc操作是，对应kdc节点上的用户，应当知道该用户的密码，并且该用户应存在于kdc节点sudoers列表内 -password = "password" #--上面提到的kdc接应用户的登录密码，用于ssh操作 diff --git a/dss-user-manager/src/main/resources/default/AddschedulerUser.sh b/dss-user-manager/src/main/resources/default/AddschedulerUser.sh deleted file mode 100644 index d9a5a02e4..000000000 --- a/dss-user-manager/src/main/resources/default/AddschedulerUser.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/usr/bin/env bash - -user=$1 -password=$2 -installDir=$3 - - -if grep -i "^${user}=" $installDir/token.properties; - then - if [ "$(uname)" == "Darwin" ]; - then - sed -i '' "s/^$user=.*/$user=$password/" $installDir/token.properties - else - sed -i "s/^$user=.*/$user=$password/" $installDir/token.properties - fi -else - echo "$user=$password" >> $installDir/token.properties -fi diff --git a/dss-user-manager/src/main/resources/default/CreateLdapAccount.sh b/dss-user-manager/src/main/resources/default/CreateLdapAccount.sh deleted file mode 100644 index 0f4c8cbe1..000000000 --- a/dss-user-manager/src/main/resources/default/CreateLdapAccount.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -source /etc/profile -ldap_user=$1 -ldap_password=$2 -ldap_script_path=$3 -source $ldap_script_path/tools/venv/bin/activate && sudo $ldap_script_path/tools/venv/bin/python $ldap_script_path/tools/bin/ldap_user.py add_with_pw $ldap_user -p $ldap_password -echo "******************LDAP USER CREATED***********************" diff --git a/dss-user-manager/src/main/resources/default/CreateLinuxUser.sh b/dss-user-manager/src/main/resources/default/CreateLinuxUser.sh deleted file mode 100644 index 32e42bccc..000000000 --- a/dss-user-manager/src/main/resources/default/CreateLinuxUser.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/bash - -source /etc/profile -server_user_strs=$1 -echo $server_user_strs -add_user_name=$2 -add_user_password=$3 -server_user_array=(${server_user_strs//,/ }) -for server_user_str in ${server_user_array[@]} -do - server_user_info=(${server_user_str//#/ }) - server_host=${server_user_info[0]} - server_user_name=${server_user_info[1]} - server_user_password=${server_user_info[2]} - echo "${server_host},${server_user_name},${server_user_password}" - - sudo sshpass -p $server_user_password ssh -o ConnectTimeout=1 $server_user_name@$server_host "echo success" - [ $? -ne 0 ] && echo "logon server:${server_host} failed" && exit 254 -done - -echo "************Start creating user*****************" - -for server_user_str in ${server_user_array[@]} -do - - server_user_info=(${server_user_str//#/ }) - server_host=${server_user_info[0]} - server_user_name=${server_user_info[1]} - server_user_password=${server_user_info[2]} - #sshpass -p $server_user_password ssh $server_user_name@$server_host "sudo useradd $add_user_name && echo $add_user_password |sudo -i passwd --stdin $add_user_name" - sshpass -p $server_user_password ssh $server_user_name@$server_host "sudo useradd $add_user_name -s /sbin/nologin" - - [ $? -ne 0 ] && echo "create user failed:${host}" && exit 254 -done diff --git a/dss-user-manager/src/main/resources/default/HdfsPath.sh b/dss-user-manager/src/main/resources/default/HdfsPath.sh deleted file mode 100644 index 1f2956501..000000000 --- a/dss-user-manager/src/main/resources/default/HdfsPath.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash -source /etc/profile -user=$1 -dir=$2 -echo $1 $2 -id $user -if [ $? -ne 0 ]; then - sudo useradd $user -s /sbin/nologin - echo "create user successfully" -fi - -hdfs dfs -mkdir -p $dir -hdfs dfs -chown $user:$user $dir diff --git a/dss-user-manager/src/main/resources/default/Kerberos.sh b/dss-user-manager/src/main/resources/default/Kerberos.sh deleted file mode 100644 index 80c3eb502..000000000 --- a/dss-user-manager/src/main/resources/default/Kerberos.sh +++ /dev/null @@ -1,146 +0,0 @@ -#!/bin/bash -source /etc/profile -#需要将当前登录用户，如dss加入到sudoers - -#函数 -check_principal_exist(){ - all_principal=`timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo /usr/sbin/kadmin.local -q \"list_principals\""` #echo "all_principal:"$all_principal - principal=$1 - if [[ $all_principal =~ $principal ]] - then - #echo "包含" - return 1 - else - #echo "不包含" - return 0 - fi -} - -add_principal(){ - principalPrefix=$1 - echo "add_principal func,principalPrefix:"$principalPrefix - check_principal_exist "$principalPrefix@$REALM" - ifexist=$? - if [ $ifexist -eq 1 ] - then - echo "已有principal" - else - echo "没有principal,将会生成" - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo /usr/sbin/kadmin.local -q \"addprinc -randkey $principalPrefix\"" - fi -} - -generate_user(){ - username=$1 - if id -u $username >/dev/null 2>&1; then - echo "user exists" - else - echo "user does not exist, so we will create!" - sudo useradd $username - fi -} - -gen_keytab(){ - user=$1 - host=$2 - principalPrefix="$user/$host" - principal="$user/$host@$REALM" - add_principal $principalPrefix - if [[ $? -ne 0 ]];then - echo "create keytab failed!!!" - exit 1 - fi - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo rm -rf /tmp/$host.$user.keytab" - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo /usr/sbin/kadmin.local -q \"xst -norandkey -k /tmp/$host.$user.keytab $user/$host\"" - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo chmod 755 /tmp/$host.$user.keytab" - timeout 30 sshpass -p $PASSWORD scp -P $SSH_PORT $KDC_USER@$KDCSERVER:/tmp/$host.$user.keytab ./ - if [[ -f "$host.$user.keytab" ]]; then - sudo mv ./$host.$user.keytab $CENTER_KEYTAB_PATH/$user.keytab - if [[ $? != 0 ]];then - echo "rename keytab failed!" - else - generate_user $user - sudo chown $user $CENTER_KEYTAB_PATH/$user.keytab - sudo su - $user -c "kinit -kt $CENTER_KEYTAB_PATH/$user.keytab $principal" - deployUser=`whoami` - sudo su - $deployUser -c "crontab -l > conf && echo '* */12 * * * sudo -u $user kinit -kt $CENTER_KEYTAB_PATH/$user.keytab $principal' >> conf && crontab conf && rm -f conf" - fi - else - echo "the $user.keytab does not exist, please check your previous steps!" - fi -} - -gen_keytab_user(){ - user=$1 - principalPrefix="$user" - principal="$user@$REALM" - add_principal $principalPrefix - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo rm -rf /tmp/$user.keytab" - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo /usr/sbin/kadmin.local -q \"xst -norandkey -k /tmp/$user.keytab $user\"" - timeout 30 sshpass -p $PASSWORD ssh -p $SSH_PORT $KDC_USER@$KDCSERVER "sudo chmod 755 /tmp/$user.keytab" - timeout 30 sshpass -p $PASSWORD scp -P $SSH_PORT $KDC_USER@$KDCSERVER:/tmp/$user.keytab ./ - if [[ -f "$user.keytab" ]]; then - sudo mv ./$user.keytab $CENTER_KEYTAB_PATH/$user.keytab - generate_user $user - sudo chown $user $CENTER_KEYTAB_PATH/$user.keytab - sudo su - $user -c "kinit -kt $CENTER_KEYTAB_PATH/$user.keytab $principal" - sudo su - op -c "crontab -l > conf && echo '* */12 * * * sudo -u $user kinit -kt $CENTER_KEYTAB_PATH/$user.keytab $principal' >> conf && crontab conf && rm -f conf" - else - echo "the $user.keytab does not exist, please check your previous steps!" - fi - -} - - - -#第一个参数为功能参数（必须有）,第二个为user（必须有），第三个为host（可以有） -if [ $# -lt 3 ] || [ $# -gt 9 ]; then - echo -e "\033[31m \033[05m请确认您的操作,输入格式如下 功能参数 [user|user hostname]\033[0m" - echo "Usage: $0 genenateKeytab {username|username hostname}" - echo `date '+%Y-%m-%d %H:%M:%S'`" parameters:"$* >>/tmp/deltaKerberos.log - exit 1 -else - if [ $# -eq 8 ]; then - user=$1 - CENTER_KEYTAB_PATH=$2 - SSH_PORT=$3 - KDCSERVER=$4 - KDC_USER=$5 - PASSWORD=$6 - REALM=$7 - KERBEROS_ENABLE=$8 - echo $user - echo $CENTER_KEYTAB_PATH - echo $SSH_PORT - echo $KDCSERVER_$KDC_USER - echo $REALM - echo $KERBEROS_ENABLE - if [ $KERBEROS_ENABLE = "0" ]; then - echo "kerberos is disabled" - else - echo "kerberos is enable" - echo `date '+%Y-%m-%d %H:%M:%S'`" in genenate_key_tab username:"$user >>/tmp/deltaKerberos.log - gen_keytab_user $user - fi - else - user=$1 - host=$2 - CENTER_KEYTAB_PATH=$3 - SSH_PORT=$4 - KDCSERVER=$5 - KDC_USER=$6 - PASSWORD=$7 - REALM=$8 - KERBEROS_ENABLE=$9 - echo $REALM - echo $KERBEROS_ENABLE - if [ $KERBEROS_ENABLE = "0" ]; then - echo "kerberos is disabled" - else - echo "kerberos1 is enable" - echo `date '+%Y-%m-%d %H:%M:%S'`" in genenate_key_tab username:"$user" hostname:"$host >>/tmp/deltaKerberos.log - gen_keytab $user $host - fi - fi -fi -exit 0 diff --git a/dss-user-manager/src/main/resources/default/LinuxPath.sh b/dss-user-manager/src/main/resources/default/LinuxPath.sh deleted file mode 100644 index 03db38d85..000000000 --- a/dss-user-manager/src/main/resources/default/LinuxPath.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash -source /etc/profile -user=$1 -dir=$2 -echo $1 $2; -id $user -if [ $? -ne 0 ]; then - useradd $user -s /sbin/nologin - echo "create user successfully" -fi -sudo mkdir -p $dir -sudo chown $user:$user $dir diff --git a/dss-user-manager/src/main/resources/default/Metastore.sh b/dss-user-manager/src/main/resources/default/Metastore.sh deleted file mode 100644 index 3d72f6658..000000000 --- a/dss-user-manager/src/main/resources/default/Metastore.sh +++ /dev/null @@ -1,44 +0,0 @@ -#!/bin/bash -source /etc/profile -user_name=$1 -db_name=$2 -path=$3 -realm=$4 -admin=$5 -ktPath=$6 -host_name=`hostname` -kerberos_enable=$7 -echo $kerberos_enable -if [ $kerberos_enable = "0" ]; then - echo "kerberos is disabled" -else - echo "kerberos is enabled" - kinit -kt $ktPath/$admin.keytab $admin/${host_name}@${realm} -fi - - #二、metastore操作 -hive -e "create database if not exists $db_name" -if [[ $? -ne 0 ]]; then - echo "create database failed!" -else - #修改数据库所属，将principal用户添加到metastore侧hive-site.xml hive.users.in.admin.role中 - if [ $kerberos_enable = "0" ]; then - hive -e "grant all on database $db_name to user $user_name" - else - hive -e "set role admin; grant all on database $db_name to user $user_name" - fi -fi - - #三、hdfs操作 -if [[ $? -ne 0 ]]; then - #回滚 - hive -e "drop database $db_name" - echo "grant database failed,rollback finished!" -else - echo "grant database $db_name successfully!" - #修改hdfs路径所属 - hdfs dfs -chown $user_name:$user_name $path - #修改hdfs路径权限 - hdfs dfs -chmod -R 700 $path - echo "hdfs operation successfully!" -fi diff --git a/dss-user-manager/src/main/resources/read_anlexander.txt b/dss-user-manager/src/main/resources/read_anlexander.txt deleted file mode 100644 index fc60487e0..000000000 --- a/dss-user-manager/src/main/resources/read_anlexander.txt +++ /dev/null @@ -1,20 +0,0 @@ -一键化开通账户 - 服务目录结构： - 进入口调用： - 入口调用：dss-user-manager\src\main\java\com\webank\wedatasphpere\dss\user\service\impl\UserAuthorizationClient.java - -#参考配置文件resource/config/properties.conf -#metastore功能使用说明 - 参数：base_path --元数据位于hdfs位置上的base - metastore_sh --执行脚本的位置 - db_tail --hdfs上数据库文件的尾缀 - realm --kerberos域 - admin --获得hive metastore服务admin角色授权的用户（如hdfs,非常重要，否则无法完成授权） - -#kerberos功能使用说明 - 参数：keytabPath --keytab在本节点上的存放位置 - shellFile --执行脚本的位置 - sshPort --ssh操作使用的端口 - kdcNode --kerberos服务kdc节点 - kdcUser --在进行ssh kdc操作是，对应kdc节点上的用户，应当知道该用户的密码，并且该用户应存在于kdc节点sudoers列表内 - password --上面提到的kdc接应用户的登录密码，用于ssh操作 \ No newline at end of file diff --git a/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/KerberosCommandTest.java b/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/KerberosCommandTest.java deleted file mode 100644 index 445b4889d..000000000 --- a/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/KerberosCommandTest.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import org.junit.jupiter.api.Test; - -import java.io.IOException; - -class KerberosCommandTest { - - @Test -// @MethodSource("body") - void authorization() { - AuthorizationBody body = new AuthorizationBody(); - body.setUsername("anlexander"); - body.setPassword("123321"); - KerberosCommand test = new KerberosCommand(); - try { - test.authorization(body); - } catch (Exception e) { - e.printStackTrace(); - } - System.out.println("当前测试方法结束"); - } -} diff --git a/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/MetastoreCommandTest.java b/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/MetastoreCommandTest.java deleted file mode 100644 index da1fcb91e..000000000 --- a/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/MetastoreCommandTest.java +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - - -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import org.junit.jupiter.api.Test; - -class MetastoreCommandTest { - - @Test -// @MethodSource("body") - void authorization() { - AuthorizationBody body = new AuthorizationBody(); - body.setUsername("anlexander"); - body.setPassword("123321"); - MetastoreCommand test = new MetastoreCommand(); - - try { - test.authorization(body); - } catch (Exception e) { - e.printStackTrace(); - } - - System.out.println("当前测试方法结束"); - } -} diff --git a/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/WorkspaceCommandTest.java b/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/WorkspaceCommandTest.java deleted file mode 100644 index 68098eaaf..000000000 --- a/dss-user-manager/src/test/java/com/webank/wedatasphpere/dss/user/service/impl/WorkspaceCommandTest.java +++ /dev/null @@ -1,46 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - - -package com.webank.wedatasphpere.dss.user.service.impl; - - -import com.webank.wedatasphpere.dss.user.conf.DSSUserManagerConfig; -import com.webank.wedatasphpere.dss.user.dto.request.AuthorizationBody; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.params.provider.MethodSource; -import org.junit.jupiter.params.provider.ValueSource; - -class WorkspaceCommandTest { - - @Test -// @MethodSource("body") - void authorization() { - AuthorizationBody body = new AuthorizationBody(); - body.setUsername("luxl"); - body.setPassword("123321"); - WorkspaceCommand test = new WorkspaceCommand(); - try { - test.authorization(body); - } catch (Exception e) { - e.printStackTrace(); - } - System.out.println("当前测试方法结束"); - -// DSSUserManagerConfig.LOCAL_USER_ROOT_PATH.getValue(); - } -} diff --git a/eventchecker-appjoint/pom.xml b/eventchecker-appjoint/pom.xml deleted file mode 100644 index bdf035e3d..000000000 --- a/eventchecker-appjoint/pom.xml +++ /dev/null @@ -1,139 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-eventchecker-appjoint - - - - org.apache.commons - commons-lang3 - 3.4 - - - - com.alibaba - druid - 1.0.28 - - - - com.webank.wedatasphere.dss - dss-appjoint-core - ${dss.version} - provided - true - - - - - - - - - - - - - - - - log4j - log4j - 1.2.17 - - - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - dss-eventchecker-appjoint - false - false - - src/main/assembly/distribution.xml - - - - - - - src/main/java - - **/*.xml - - - - src/main/resources - - **/*.properties - **/application.yml - **/bootstrap.yml - **/log4j2.xml - - - - - - - \ No newline at end of file diff --git a/eventchecker-appjoint/src/main/assembly/distribution.xml b/eventchecker-appjoint/src/main/assembly/distribution.xml deleted file mode 100644 index 01d8fc345..000000000 --- a/eventchecker-appjoint/src/main/assembly/distribution.xml +++ /dev/null @@ -1,136 +0,0 @@ - - - - dss-eventchecker-appjoint - - zip - - true - eventchecker - - - - - - lib - true - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ${basedir}/src/main/resources - - appjoint.properties - - 0777 - / - unix - - - - ${basedir}/src/main/resources - - log4j.properties - log4j2.xml - - 0777 - conf - unix - - - - - - diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/adapter/EventCheckAdapter.java b/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/adapter/EventCheckAdapter.java deleted file mode 100644 index de4025674..000000000 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/adapter/EventCheckAdapter.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.adapter; - -import org.apache.log4j.Logger; - -import java.util.Properties; - -/** - * @author georgeqiao - * @Title: EventCheckReceiver - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ -public interface EventCheckAdapter { - - boolean sendMsg(int jobId, Properties props, Logger log); - - boolean reciveMsg(int jobId, Properties props, Logger log); - -} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/entity/EventChecker.java b/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/entity/EventChecker.java deleted file mode 100644 index 2159eae25..000000000 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/entity/EventChecker.java +++ /dev/null @@ -1,277 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity; - -import com.google.gson.Gson; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionState; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution.EventCheckerNodeExecutionAction; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service.AbstractEventCheck; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service.EventCheckerService; -import org.apache.commons.lang3.StringUtils; -import org.apache.log4j.Logger; -import java.util.HashMap; -import java.util.Map; -import java.util.Properties; -import java.util.regex.Matcher; -import java.util.regex.Pattern; - -/** - * @author georgeqiao - * @Title: EventChecker - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ -public class EventChecker implements Runnable{ - public final static String WAIT_TIME = "max.receive.hours"; - public final static String WAIT_FOR_TIME = "wait.for.time"; - public final static String QUERY_FREQUENCY = "query.frequency"; - public final static String MSGTYPE="msg.type"; - public final static String SENDER="msg.sender"; - public final static String RECEIVER="msg.receiver"; - public final static String TOPIC="msg.topic"; - public final static String MSGNAME="msg.name"; - public final static String MSG="msg.body"; - public final static String EXEC_ID = "azkaban.flow.execid"; - public final static String SAVE_KEY="msg.savekey"; - public final static String USER_TIME="msg.init.querytime"; - public final static String TODAY="only.receive.today"; - public final static String AFTERSEND="msg.after.send"; - - private Properties p; - private String jobId; - private int execId; - private EventCheckerService wbDao=null; - EventCheckerNodeExecutionAction backAction = null; - public Long maxWaitTime; - public int queryFrequency; - - private static Pattern pattern = Pattern.compile("[a-zA-Z_0-9@\\-]+"); - - private static final Logger logger = Logger.getRootLogger(); - - public EventChecker(String jobName, Properties p, EventCheckerNodeExecutionAction action) { - this.p = p; -// this.jobId = p.getProperty(EXEC_ID); - this.jobId = "1"; - backAction = action; - String waitTime = p.getProperty(EventChecker.WAIT_TIME, "1"); - Double doubleWaitTime = Double.valueOf(waitTime) * 3600 * 1000; - maxWaitTime = Long.valueOf(doubleWaitTime.longValue()); - String query_frequency = p.getProperty(EventChecker.QUERY_FREQUENCY, "30000"); - queryFrequency = Integer.valueOf(query_frequency); - } - - public void run() { - try { - backAction.setState(NodeExecutionState.Running); -// getPid(); - if (p == null) { - throw new RuntimeException("Properties is null. Can't continue"); - } - if (checkParamMap(p, MSGTYPE)) { - throw new RuntimeException("parameter " + MSGTYPE + " can not be blank."); - } - if (checkParamMap(p, TOPIC)) { - throw new RuntimeException("parameter " + TOPIC + " can not be blank."); - } else { - String topic = p.getProperty(TOPIC); - if (!topic.matches("[^_]*_[^_]*_[^_]*")) { - throw new RuntimeException("Error format of topic parameter. Accept: XX_XX_XX."); - } - } - if (checkParamMap(p, MSGNAME)) { - throw new RuntimeException("parameter " + MSGNAME + " can not be blank."); - } - wbDao = EventCheckerService.getInstance(); - execId = Integer.parseInt(jobId); - boolean success = false; - if (p.getProperty(MSGTYPE).equals("SEND")) { - if (checkParamMap(p, SENDER)) { - throw new RuntimeException("parameter " + SENDER + " can not be blank."); - } else { - String sender = p.getProperty(SENDER); - if (!sender.matches("[^@]*@[^@]*@[^@]*")) { - throw new RuntimeException("Error format of sender parameter. Accept: XX@XX@XX."); - } - } -// if (checkParamMap(p, MSG)) { -// throw new RuntimeException("Must specify a " + MSG -// + " key and value."); -// } - if (p.containsKey(MSG) && StringUtils.isNotEmpty(p.getProperty(MSG)) && p.getProperty(MSG).length() > 250) { - throw new RuntimeException("parameter " + MSG + " length less than 250 !"); - } - success = wbDao.sendMsg(execId, p, logger); - if (success) { - backAction.setState(NodeExecutionState.Success); - - } else { - throw new RuntimeException("Failed Send message."); - } - }else if(p.getProperty(MSGTYPE).equals("RECEIVE")) { - backAction.eventType("RECEIVE"); - receiveMsg(); - } else - { - throw new RuntimeException("Please input correct parameter of msg.type, Select RECEIVE Or SEND."); - } - }catch (Exception ex){ - backAction.setState(NodeExecutionState.Failed); - throw ex; - } - - } - - public boolean receiveMsg(){ - boolean success = false; - if(p.getProperty(MSGTYPE).equals("RECEIVE")) { - if (checkParamMap(p, RECEIVER)) { - backAction.setState(NodeExecutionState.Failed); - throw new RuntimeException("parameter " + RECEIVER + " can not be blank."); - } else { - String receiver = p.getProperty(RECEIVER); - if (!receiver.matches("[^@]*@[^@]*@[^@]*")) { - backAction.setState(NodeExecutionState.Failed); - throw new RuntimeException("Error format of receiver parameter. Accept: XX@XX@XX."); - } - } -// checkTimeParam(p, WAIT_FOR_TIME); - String userTime = checkTimeParamMap(p, USER_TIME); - if (StringUtils.isNotEmpty(userTime)) { - p.put(USER_TIME, userTime); - } - success = wbDao.reciveMsg(execId, p, logger); - if (success) { - backAction.saveKeyAndValue(getJobSaveKeyAndValue()); - backAction.setState(NodeExecutionState.Success); - } else { - backAction.setState(NodeExecutionState.Running); - } - } - return success; - } - - public String getJobSaveKeyAndValue(){ - Map saveValueMap = new HashMap<>(); - String msgBody=p.getProperty(MSG, "{}"); - String saveKey=p.getProperty(SAVE_KEY,"msg.body"); - saveValueMap.put(saveKey, msgBody); - Gson gson = new Gson(); - String saveValueJson = gson.toJson(saveValueMap); - logger.info("Output msg body: "+saveValueJson); - return saveValueJson; - } - - public void cancel() throws InterruptedException { -// AbstractEventCheck.closeDruidDataSource(); -// throw new RuntimeException("Kill this eventchecker."); - } - - private boolean checkParamMap(Properties p, String key){ - boolean checkFlag = false; - if(!p.containsKey(key)){ - throw new RuntimeException("parameter " + key + " is Empty."); - } - if(p.containsKey(key)){ - if(StringUtils.isEmpty(p.getProperty(key))){ - checkFlag = true; - } - } - if(!MSG.equals(key) && StringUtils.contains(p.getProperty(key), " ")){ - throw new RuntimeException("parameter " + key + " can not contains space !"); - } - if(!checkNoStandardStr(p.getProperty(key))){ - throw new RuntimeException("parameter " + key + " Accept letter and number and _@- only."); - } - if(p.getProperty(key).length() > 45){ - throw new RuntimeException("parameter " + key + " length less than 45 !"); - } - return checkFlag; - } - - private boolean checkNoStandardStr(String param){ - Matcher matcher = pattern.matcher(param); - return matcher.matches(); - } - - private void checkTimeParam(Properties p, String key){ - if(p.containsKey(key)){ - String waitForTime= p.getProperty(key); - if(!waitForTime.matches("^(0?[0-9]|1[0-9]|2[0-3]):(0?[0-9]|[1-5][0-9])$")){ - throw new RuntimeException("Parameter " + key + " Time format error ! For example: HH:mm"); - } - } - } - - private String checkTimeParamMap(Properties p, String key){ - if(p.containsKey(key)){ - String userTime = p.getProperty(key); - Pattern ptime = Pattern.compile("^([1][7-9][0-9][0-9]|[2][0][0-9][0-9])(\\-)([0][1-9]|[1][0-2])(\\-)([0-2][1-9]|[3][0-1])(\\s)([0-1][0-9]|[2][0-3])(:)([0-5][0-9])(:)([0-5][0-9])$"); - Matcher m = ptime.matcher(userTime); - if(!m.matches()){ - throw new RuntimeException("Parameter " + key + " Time format error ! For example: yyyy-MM-dd HH:mm:ss"); - } - return userTime; - }else{ - return null; - } - } - -// private String getPid(){ -// // get name representing the running Java virtual machine. -// String name = ManagementFactory.getRuntimeMXBean().getName(); -// System.out.println(name); -// // get pid -// String pid = name.split("@")[0]; -// logger.info("EventCheck Pid is:" + pid); -// return pid; -// } - - public static void main(String[] args) { - Properties p = new Properties(); - p.put("azkaban.flow.execid","111"); - - p.put("msg.type","RECEIVE"); - p.put("msg.receiver","project@job@v_zhu"); - p.put("msg.topic","bdp_new_test"); - p.put("msg.name","TestCheck"); - p.put("msg.savekey","msg.body"); - p.put("query.frequency","60"); - p.put("wait.time","1"); - p.put("only.receive.today","true"); - p.put("msg.after.send","true"); -// p.put("wait.for.time","22:00"); - -// p.put("msg.type","SEND"); -// p.put("msg.sender","project@job@v_zhu"); -// p.put("msg.topic","bdp_new_test"); -// p.put("msg.name","TestCheck"); -// p.put("msg.body","msg.body"); - - p.put("msg.eventchecker.jdo.option.name","msg"); - p.put("msg.eventchecker.jdo.option.url","jdbc:mysql://127.0..0.1:3306/wtss_qyh_test?useUnicode=true&characterEncoding=UTF-8"); - p.put("msg.eventchecker.jdo.option.username","root"); - p.put("msg.eventchecker.jdo.option.password","MTIzNDU2"); - -// EventChecker ec = new EventChecker("AA",p); -// ec.run(); - } - - -} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/AbstractEventCheck.java b/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/AbstractEventCheck.java deleted file mode 100644 index 3cecbb0e1..000000000 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/AbstractEventCheck.java +++ /dev/null @@ -1,207 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service; - -import com.alibaba.druid.pool.DruidDataSource; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.adapter.EventCheckAdapter; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.connector.EventDruidFactory; -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.EventChecker; - -import org.apache.log4j.Logger; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.lang.management.ManagementFactory; -import java.net.InetAddress; -import java.net.NetworkInterface; -import java.net.SocketException; -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.ArrayList; -import java.util.Enumeration; -import java.util.List; -import java.util.Properties; - -import javax.sql.DataSource; - -/** - * @author georgeqiao - * @Title: AbstractEventCheck - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ -public abstract class AbstractEventCheck implements EventCheckAdapter { - static DataSource msgDS; - String topic; - String msgName; - String receiver; - String sender; - String receiveToday; - String userTime; - String waitTime; - String query_frequency; - String wait_for_time; - String msg; - String afterSend; - - DataSource getMsgDS(Properties props, Logger log){ - if (msgDS == null) { - msgDS = EventDruidFactory.getMsgInstance(props, log); - if (msgDS == null) { - log.error("Error getting Druid DataSource instance"); - } - } - return msgDS; - } - - void initECParams(Properties props){ - topic = props.getProperty(EventChecker.TOPIC); - msgName = props.getProperty(EventChecker.MSGNAME); - receiver = props.getProperty(EventChecker.RECEIVER); - sender = props.getProperty(EventChecker.SENDER); - msg = props.getProperty(EventChecker.MSG); - receiveToday = props.getProperty(EventChecker.TODAY); - userTime = props.getProperty(EventChecker.USER_TIME); - waitTime = props.getProperty(EventChecker.WAIT_TIME, "1"); - query_frequency = props.getProperty(EventChecker.QUERY_FREQUENCY, "30000"); -// wait_for_time = props.getProperty(EventChecker.WAIT_FOR_TIME); - afterSend = props.getProperty(EventChecker.AFTERSEND); - } - - Connection getEventCheckerConnection(Properties props, Logger log){ - Connection connection = null; - try { - connection = getMsgDS(props,log).getConnection(); - } catch (SQLException e) { - throw new RuntimeException("Error getting DB Connection instance {} " + e); - } - return connection; - } - - @Override - public boolean sendMsg(int jobId, Properties props, Logger log) { - return false; - } - - @Override - public boolean reciveMsg(int jobId, Properties props, Logger log) { - return false; - } - - void closeConnection(Connection conn, Logger log) { - if (conn != null) { - try { - conn.close(); - } catch (SQLException e) { - log.error("Error closing connection", e); - } - } - } - - void closeQueryRef(ResultSet rs, Logger log) { - if (rs != null) { - try { - rs.close(); - } catch (SQLException e) { - log.error("Error closing result set", e); - } - } - - } - - void closeQueryStmt(PreparedStatement stmt, Logger log) { - if (stmt != null) { - try { - stmt.close(); - } catch (SQLException e) { - log.error("Error closing result stmt", e); - } - } - - } - - - public static void closeDruidDataSource() { - DruidDataSource msgDSObject = (DruidDataSource) msgDS; - if (msgDSObject != null) { - msgDSObject.close(); - } - - } - - String getLinuxLocalIp(Logger log) { - String ip = "127.0.0.1"; - try { - for (Enumeration en = NetworkInterface.getNetworkInterfaces(); en.hasMoreElements(); ) { - NetworkInterface intf = en.nextElement(); - String name = intf.getName(); - if (!name.contains("docker") && !name.contains("lo")) { - for (Enumeration enumIpAddr = intf.getInetAddresses(); enumIpAddr.hasMoreElements(); ) { - InetAddress inetAddress = enumIpAddr.nextElement(); - if (!inetAddress.isLoopbackAddress()) { - String ipaddress = inetAddress.getHostAddress().toString(); - if (!ipaddress.contains("::") && !ipaddress.contains("0:0:") && !ipaddress.contains("fe80")) { - ip = ipaddress; - } - } - } - } - } - } catch (SocketException ex) { - log.warn("get ip failed", ex); - - } - log.info("Send IP:" + ip); - return ip; - } - -// static void AzkabanStatusMonitor() { -// String name = ManagementFactory.getRuntimeMXBean().getName(); -// System.out.println(name); -// String pid = name.split("@")[0]; -// Process process = null; -// List processList = new ArrayList(); -// InputStream in = null; -// String[] cmds = {"/bin/sh", "-c", "ps -ef | grep " + pid + " | grep -v 'grep' | awk '{print $3}'"}; -// try { -// process = Runtime.getRuntime().exec(cmds); -// process.waitFor(); -// in = process.getInputStream(); -// BufferedReader reader = new BufferedReader(new InputStreamReader(in)); -// String line; -// while ((line = reader.readLine()) != null) { -// processList.add(line); -// } -// reader.close(); -// } catch (IOException e) { -// e.printStackTrace(); -// } catch (InterruptedException e) { -// e.printStackTrace(); -// } -// -// for (String ppid : processList) { -// System.out.println(ppid); -// -// } -// } -} diff --git a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/AbstractEventCheckReceiver.java b/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/AbstractEventCheckReceiver.java deleted file mode 100644 index fbd1c887a..000000000 --- a/eventchecker-appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/service/AbstractEventCheckReceiver.java +++ /dev/null @@ -1,225 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.service; - -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.EventChecker; - -import org.apache.commons.lang3.StringUtils; -import org.apache.commons.lang3.time.DateFormatUtils; -import org.apache.log4j.Logger; - -import java.sql.Connection; -import java.sql.PreparedStatement; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.util.Date; -import java.util.Properties; - -/** - * @author georgeqiao - * @Title: AbstractEventCheckReceiver - * @ProjectName Azkaban-EventChecker - * @date 2019/9/1822:10 - * @Description: TODO - */ -public class AbstractEventCheckReceiver extends AbstractEventCheck{ - /** - * Fill the result into the source - * @param props - * @param log - * @param consumedMsgInfo - * @return - */ - String setConsumedMsg(Properties props, Logger log, String[] consumedMsgInfo){ - String vNewMsgID = ""; - try { - if(consumedMsgInfo!=null && consumedMsgInfo.length == 4){ - vNewMsgID = consumedMsgInfo[0]; - String vMsgName = consumedMsgInfo[1]; - String vSender = consumedMsgInfo[2]; - String vMsg = consumedMsgInfo[3]; - if (null == vMsg) { - props.put(EventChecker.MSG, "NULL"); - } else { - props.put(EventChecker.MSG, vMsg); - } - log.info("Received message : messageID: " + vNewMsgID + ", messageName: " + vMsgName + ", receiver: " + vSender - + ", messageBody: " + vMsg); - } - }catch (Exception e) { - log.error("Error set consumed message failed {} setConsumedMsg failed" + e); - return vNewMsgID; - } - return vNewMsgID; - } - - /** - * Update consumption status - * @param jobId - * @param props - * @param log - * @param consumedMsgInfo - * @return - */ - boolean updateMsgOffset(int jobId, Properties props, Logger log, String[] consumedMsgInfo,String lastMsgId){ - boolean result = false; - String vNewMsgID = "-1"; - PreparedStatement updatePstmt = null; - PreparedStatement pstmtForGetID = null; - Connection msgConn = null; - vNewMsgID = setConsumedMsg(props,log,consumedMsgInfo); - try { - if(StringUtils.isNotEmpty(vNewMsgID) && StringUtils.isNotBlank(vNewMsgID) && !"-1".equals(vNewMsgID)){ - msgConn = getEventCheckerConnection(props,log); - if(msgConn == null) return false; - msgConn.setAutoCommit(false); - String sqlForReadMsgID = "SELECT msg_id FROM event_status WHERE receiver=? AND topic=? AND msg_name=? for update"; - pstmtForGetID = msgConn.prepareCall(sqlForReadMsgID); - pstmtForGetID.setString(1, receiver); - pstmtForGetID.setString(2, topic); - pstmtForGetID.setString(3, msgName); - ResultSet rs = pstmtForGetID.executeQuery(); - String nowLastMsgId = rs.last()==true ? rs.getString("msg_id"):"0"; - log.info("receive message successfully , Now check to see if the latest offset has changed ,nowLastMsgId is {} " + nowLastMsgId); - if("0".equals(nowLastMsgId) || nowLastMsgId.equals(lastMsgId)){ - - int vProcessID = jobId; - String vReceiveTime = DateFormatUtils.format(new Date(), "yyyy-MM-dd HH:mm:ss");; - String sqlForUpdateMsg = "INSERT INTO event_status(receiver,topic,msg_name,receive_time,msg_id) VALUES(?,?,?,?,?) ON DUPLICATE KEY UPDATE receive_time=VALUES(receive_time),msg_id= CASE WHEN msg_id= " + lastMsgId + " THEN VALUES(msg_id) ELSE msg_id END"; - log.info("last message offset {} is:" + lastMsgId); - updatePstmt = msgConn.prepareCall(sqlForUpdateMsg); - updatePstmt.setString(1, receiver); - updatePstmt.setString(2, topic); - updatePstmt.setString(3, msgName); - updatePstmt.setString(4, vReceiveTime); - updatePstmt.setString(5, vNewMsgID); - int updaters = updatePstmt.executeUpdate(); - log.info("updateMsgOffset successful {} update result is:" + updaters); - if(updaters != 0){ - log.info("Received message successfully , update message status succeeded, consumed flow execution ID: " + vProcessID); - //return true after update success - result = true; - }else{ - log.info("Received message successfully , update message status failed, consumed flow execution ID: " + vProcessID); - result = false; - } - }else{ - log.info("the latest offset has changed , Keep waiting for the signal"); - result = false; - } - msgConn.commit(); - }else{ - result = false; - } - }catch (SQLException e){ - log.error("Error update Msg Offset" + e); - try { - msgConn.rollback(); - } catch (SQLException ex) { - log.error("transaction rollback failed " + e); - } - return false; - }finally { - closeQueryStmt(pstmtForGetID, log); - closeQueryStmt(updatePstmt, log); - closeConnection(msgConn, log); - } - return result; - } - - /** - * get consumption progress - * @param jobId - * @param props - * @param log - * @return - */ - String getOffset(int jobId, Properties props, Logger log){ - String sqlForReadMsgID = "SELECT msg_id FROM event_status WHERE receiver=? AND topic=? AND msg_name=?"; - PreparedStatement pstmtForGetID = null; - Connection msgConn = null; - ResultSet rs = null; - boolean flag = false; - String lastMsgId = "0"; - try { - msgConn = getEventCheckerConnection(props,log); - pstmtForGetID = msgConn.prepareCall(sqlForReadMsgID); - pstmtForGetID.setString(1, receiver); - pstmtForGetID.setString(2, topic); - pstmtForGetID.setString(3, msgName); - rs = pstmtForGetID.executeQuery(); - lastMsgId = rs.last()==true ? rs.getString("msg_id"):"0"; - } catch (SQLException e) { - throw new RuntimeException("get Offset failed " + e); - }finally { - closeQueryStmt(pstmtForGetID,log); - closeConnection(msgConn,log); - closeQueryRef(rs,log); - } - log.info("The last record id was " + lastMsgId); - return lastMsgId; - } - - /** - * Consistent entrance to consumer message - * @param jobId - * @param props - * @param log - * @param params params[startQueryTime,endQueryTime,vMsgID] - * @return - */ - String[] getMsg(Properties props, Logger log,String ... params){ - String sqlForReadTMsg = "SELECT * FROM event_queue WHERE topic=? AND msg_name=? AND send_time >=? AND send_time <=? AND msg_id >? ORDER BY msg_id ASC LIMIT 1"; - PreparedStatement pstmt = null; - Connection msgConn = null; - ResultSet rs = null; - String[] consumedMsgInfo = null; - try { - msgConn = getEventCheckerConnection(props,log); - pstmt = msgConn.prepareCall(sqlForReadTMsg); - pstmt.setString(1, topic); - pstmt.setString(2, msgName); - pstmt.setString(3, params[0]); - pstmt.setString(4, params[1]); - pstmt.setString(5, params[2]); - log.info("param {} StartTime: " + params[0] + ", EndTime: " + params[1] - + ", Topic: " + topic + ", MessageName: " + msgName + ", LastMessageID: " + params[2]); - rs = pstmt.executeQuery(); - - if(rs.last()){ - consumedMsgInfo = new String[4]; - String[] msgKey = new String[]{"msg_id","msg_name","sender","msg"}; - for (int i = 0;i <= 3;i++) { - consumedMsgInfo[i] = rs.getString(msgKey[i]); - } - } - } catch (SQLException e) { - throw new RuntimeException("EventChecker failed to receive message" + e); - } finally { - closeQueryStmt(pstmt, log); - closeConnection(msgConn, log); - closeQueryRef(rs, log); - } - return consumedMsgInfo; - } - - @Override - public boolean reciveMsg(int jobId, Properties props, Logger log) { - return super.reciveMsg(jobId, props, log); - } -} diff --git a/eventchecker-appjoint/src/main/resources/appjoint.properties b/eventchecker-appjoint/src/main/resources/appjoint.properties deleted file mode 100644 index 86c926c42..000000000 --- a/eventchecker-appjoint/src/main/resources/appjoint.properties +++ /dev/null @@ -1,24 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -msg.eventchecker.jdo.option.name=msg -msg.eventchecker.jdo.option.url=jdbc:mysql://127.0..0.1:3306/ -msg.eventchecker.jdo.option.username=user -msg.eventchecker.jdo.option.password= - - - diff --git a/eventchecker-appjoint/src/main/resources/log4j.properties b/eventchecker-appjoint/src/main/resources/log4j.properties deleted file mode 100644 index 0807e6087..000000000 --- a/eventchecker-appjoint/src/main/resources/log4j.properties +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO diff --git a/eventchecker-appjoint/src/main/resources/log4j2.xml b/eventchecker-appjoint/src/main/resources/log4j2.xml deleted file mode 100644 index 3923cd9f3..000000000 --- a/eventchecker-appjoint/src/main/resources/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerAppJoint.scala b/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerAppJoint.scala deleted file mode 100644 index a9c6b0e4e..000000000 --- a/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerAppJoint.scala +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution - -import java.util - -import com.webank.wedatasphere.dss.appjoint.AppJoint -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl - -/** - * Created by enjoyyin on 2019/11/5. - */ -class EventCheckerAppJoint extends AppJointUrlImpl with AppJoint { - - private var params: util.Map[String, AnyRef] = _ - private var nodeExecution: NodeExecution = _ - - override def getAppJointName: String = "EventChecker" - - override def init(baseUrl: String, params: util.Map[String, AnyRef]): Unit = { - setBaseUrl(baseUrl) - this.params = params - } - - override def getNodeExecution: NodeExecution = { - if(nodeExecution == null) synchronized { - if(nodeExecution == null) { - nodeExecution = new EventCheckerNodeExecution() - nodeExecution.setBaseUrl(getBaseUrl) - nodeExecution.init(params) - } - } - nodeExecution - } -} diff --git a/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerNodeExecution.scala b/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerNodeExecution.scala deleted file mode 100644 index 36876ee36..000000000 --- a/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerNodeExecution.scala +++ /dev/null @@ -1,165 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution; - -import java.util -import java.util.{Properties, UUID} - -import com.webank.wedatasphere.dss.appjoint.execution.common._ -import com.webank.wedatasphere.dss.appjoint.execution.core._ -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.EventChecker -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrl -import com.webank.wedatasphere.dss.appjoint.service.session.Session -import com.webank.wedatasphere.linkis.common.utils.Utils -import com.webank.wedatasphere.linkis.storage.LineRecord -import org.apache.commons.io.IOUtils - - - -/** - * Created by allenlliu on 2019/11/11. - */ -class EventCheckerNodeExecution extends LongTermNodeExecution with AppJointUrl with Procedure with Killable{ - import scala.collection.JavaConversions.mapAsScalaMap - var appjointParams : scala.collection.mutable.Map[String,AnyRef]= null - - /** - * 表示任务能否提交到该AppJoint去执行 - * - * @param node AppJointNode - * @return true is ok while false is not - */ - override def canExecute(node: AppJointNode, context: NodeContext, session: Session): Boolean = node.getNodeType.toLowerCase.contains("eventchecker") - - - - override def progress(action: NodeExecutionAction): Float = { - //temp set - 0.5f - } - - override def log(action: NodeExecutionAction): String = { - action match { - case action: EventCheckerNodeExecutionAction => { - if (!action.state.isCompleted) { - "EventChecker is sending or waiting for message" - }else{ - "EventChecker successfully received or sent message" - } - } - } - } - - override def kill(action: NodeExecutionAction): Boolean = action match { - case longTermAction: EventCheckerNodeExecutionAction => - getScheduler.removeAsyncResponse(longTermAction) - true - } - - protected def putErrorMsg(errorMsg: String, t: Throwable, action: EventCheckerNodeExecutionAction): EventCheckerNodeExecutionAction = t match { - - case t: Exception => - val response = action.response - response.setErrorMsg(errorMsg) - response.setException(t) - response.setIsSucceed(false) - action - } - - override def init(params: util.Map[String, AnyRef]): Unit = { - this.appjointParams = params - } - - override def submit(node: AppJointNode, context: NodeContext, session:Session): NodeExecutionAction = { - val nodeAction = new EventCheckerNodeExecutionAction() - nodeAction.setId(UUID.randomUUID().toString()) - val jobName = node.getName - val scalaParams: scala.collection.mutable.Map[String,Object] =context.getRuntimeMap - val properties = new Properties() - this.appjointParams.foreach{ - case (key: String, value: Object) => properties.put(key, value.toString) - } - scalaParams.foreach { case (key: String, value: Object) => - properties.put(key, value.toString) - } - val ec = new EventChecker(jobName,properties,nodeAction) - ec.run() - nodeAction.setEc(ec) - nodeAction - } - - override def state(action: NodeExecutionAction): NodeExecutionState = { - action match { - case action: EventCheckerNodeExecutionAction => { - if (action.state.isCompleted) return action.state - if(action.eventType.equals("RECEIVE")){ - action.ec.receiveMsg() - } - action.state - } - } - } - - override def result(action: NodeExecutionAction, nodeContext: NodeContext): CompletedNodeExecutionResponse = { - val response = new CompletedNodeExecutionResponse - action match { - case action:EventCheckerNodeExecutionAction =>{ - if(action.state.equals(NodeExecutionState.Success)) { - if(action.saveKeyAndValue != null){ - val resultSetWriter = nodeContext.createTextResultSetWriter() - Utils.tryFinally { - resultSetWriter.addMetaData(null) - resultSetWriter.addRecord(new LineRecord(action.saveKeyAndValue)) - }(IOUtils.closeQuietly(resultSetWriter)) - } - response.setIsSucceed(true) - }else{ - response.setIsSucceed(false) - } - response - } - case _ => - response.setIsSucceed(false); - response - } - - } - - - private var baseUrl:String ="" - - override def getBaseUrl: String = baseUrl - - override def setBaseUrl(basicUrl: String): Unit = { - this.baseUrl = basicUrl - } - - override def createAsyncNodeExecutionResponse(node: AppJointNode, context: NodeContext, action: NodeExecutionAction): AsyncNodeExecutionResponse = { - action match { - case action: EventCheckerNodeExecutionAction => { - val response = new AsyncNodeExecutionResponse - response.setAction(action) - response.setAppJointNode(node) - response.setNodeContext(context) - response.setMaxLoopTime(action.ec.maxWaitTime) - response.setAskStatePeriod(action.ec.queryFrequency) - response - } - } - } -} diff --git a/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerNodeExecutionAction.scala b/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerNodeExecutionAction.scala deleted file mode 100644 index 0a92f9a9e..000000000 --- a/eventchecker-appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/schedulis/jobtype/execution/EventCheckerNodeExecutionAction.scala +++ /dev/null @@ -1,66 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.execution - -import com.webank.wedatasphere.dss.appjoint.execution.common.{AbstractNodeExecutionAction, CompletedNodeExecutionResponse, LongTermNodeExecutionAction, NodeExecutionState} -import com.webank.wedatasphere.dss.appjoint.schedulis.jobtype.entity.EventChecker - -/** - * Created by allenlliu on 2019/11/12. - */ -class EventCheckerNodeExecutionAction extends AbstractNodeExecutionAction with LongTermNodeExecutionAction { - private[this] var _state: NodeExecutionState = _ - private var schedulerId: Int = _ - - def state: NodeExecutionState = _state - - def setState(value: NodeExecutionState): Unit = { - _state = value - } - - - val response = new CompletedNodeExecutionResponse - - private[this] var _saveKeyAndValue: String = null - - def saveKeyAndValue: String = _saveKeyAndValue - - def saveKeyAndValue(value: String): Unit = { - _saveKeyAndValue = value - } - - private[this] var _eventType: String = "SEND" - - def eventType: String = _eventType - - def eventType(value: String): Unit = { - _eventType = value - } - - private[this] var _ec: EventChecker = null - - def ec: EventChecker = _ec - - def setEc(value: EventChecker): Unit = { - _ec = value - } - - override def setSchedulerId(schedulerId: Int): Unit = this.schedulerId = schedulerId - - override def getSchedulerId: Int = schedulerId -} diff --git a/plugins/azkaban/linkis-jobtype/bin/install.sh b/plugins/azkaban/linkis-jobtype/bin/install.sh index ff10c0612..bd29452fd 100644 --- a/plugins/azkaban/linkis-jobtype/bin/install.sh +++ b/plugins/azkaban/linkis-jobtype/bin/install.sh @@ -38,15 +38,16 @@ fi echo "start to subsitution conf" sed -i "s#jobtype.lib.dir.*#jobtype.lib.dir=$AZKABAN_JOBTYPE_DIR/linkis/lib#g" ${workDir}/private.properties -sed -i "s#wds.linkis.gateway.url.*#wds.linkis.gateway.url=$LINKIS_GATEWAY_URL#g" ${workDir}/plugin.properties +sed -i "s#wds.linkis.gateway.url.v0.*#wds.linkis.gateway.url.v0=$LINKIS_GATEWAY_URL#g" ${workDir}/plugin.properties +sed -i "s#wds.linkis.gateway.url.v1.*#wds.linkis.gateway.url.v1=$LINKIS_GATEWAY_URL#g" ${workDir}/plugin.properties sed -i "s#wds.linkis.client.flow.author.user.token.*#wds.linkis.client.flow.author.user.token=$LINKIS_GATEWAY_TOKEN#g" ${workDir}/plugin.properties isSuccess "subsitution conf" echo "$COPY Plugin" -ssh -p $SSH_PORT $AZKABAN_EXECUTOR_HOST "cd $AZKABAN_JOBTYPE_DIR;rm -rf linkis-bak; mv -f linkis ../linkis-bak" +##ssh -p $SSH_PORT $AZKABAN_EXECUTOR_HOST "cd $AZKABAN_JOBTYPE_DIR;rm -rf linkis-bak; mv -f linkis ../linkis-bak" scp -P $SSH_PORT -r ${workDir} $AZKABAN_EXECUTOR_HOST:$AZKABAN_JOBTYPE_DIR echo "reload jobType" -curl $AZKABAN_EXECUTOR_URL \ No newline at end of file +##curl $AZKABAN_EXECUTOR_URL \ No newline at end of file diff --git a/plugins/azkaban/linkis-jobtype/pom.xml b/plugins/azkaban/linkis-jobtype/pom.xml index 9d0d787b5..9c8dfd16a 100644 --- a/plugins/azkaban/linkis-jobtype/pom.xml +++ b/plugins/azkaban/linkis-jobtype/pom.xml @@ -1,9 +1,8 @@ + + + + dss + com.webank.wedatasphere.dss + 1.0.0 + + 4.0.0 + + dss-gateway-support + + + com.webank.wedatasphere.linkis + linkis-gateway-server-support + ${linkis.version} + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + + net.alchim31.maven + scala-maven-plugin + + + org.apache.maven.plugins + maven-jar-plugin + + + + + + \ No newline at end of file diff --git a/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSGatewayConfiguration.scala b/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSGatewayConfiguration.scala new file mode 100644 index 000000000..b5d30e0a9 --- /dev/null +++ b/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSGatewayConfiguration.scala @@ -0,0 +1,32 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.gateway.parser + +import com.webank.wedatasphere.linkis.common.conf.CommonVars + + +object DSSGatewayConfiguration { + val DSS_SPRING_NAME = CommonVars("wds.linkis.dss.name", "dss-server") + + val DSS_URL_LABEL_PREFIX = CommonVars("wds.dss.gateway.url.prefix.name", "labels") + + val DSS_URL_ROUTE_LABEL_PREFIX = CommonVars("wds.dss.gateway.url.prefix.name", "labelsRoute") + + val DSS_URL_APPCONNS = CommonVars("wds.dss.gateway.url.appconns", "visualis") + + +} diff --git a/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSGatewayParser.scala b/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSGatewayParser.scala new file mode 100644 index 000000000..16e6c015b --- /dev/null +++ b/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSGatewayParser.scala @@ -0,0 +1,209 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.gateway.parser + +import java.util + +import com.webank.wedatasphere.linkis.common.ServiceInstance +import com.webank.wedatasphere.linkis.gateway.exception.TooManyServiceException +import com.webank.wedatasphere.linkis.gateway.http.GatewayContext +import com.webank.wedatasphere.linkis.gateway.springcloud.SpringCloudGatewayConfiguration.{API_URL_PREFIX, normalPath} +import com.webank.wedatasphere.linkis.manager.label.builder.factory.LabelBuilderFactoryContext +import com.webank.wedatasphere.linkis.manager.label.entity.Label +import com.webank.wedatasphere.linkis.manager.label.entity.route.RouteLabel +import com.webank.wedatasphere.linkis.protocol.constants.TaskConstant +import com.webank.wedatasphere.linkis.rpc.sender.SpringCloudFeignConfigurationCache +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ + + +@Component +class DSSGatewayParser extends AbstractGatewayParser { + + val appConns = DSSGatewayConfiguration.DSS_URL_APPCONNS.getValue.split(",") + + override def shouldContainRequestBody(gatewayContext: GatewayContext): Boolean = { + var contentType = gatewayContext.getRequest.getHeaders.get("Content-Type") + if(null == contentType) { + contentType = gatewayContext.getRequest.getHeaders.get("content-type") + } + + if(contentType != null && contentType.nonEmpty + && contentType(0).contains("form-data")){ + logger.info("DSS gateway get request type is form-data") + return false + } + + gatewayContext.getRequest.getRequestURI match { + case DSSGatewayParser.DSS_URL_DEFAULT_REGEX(_, _) => true + case DSSGatewayParser.DSS_URL_REGEX(_, _,_) => true + case DSSGatewayParser.APPCONN_URL_DEFAULT_REGEX(_,appconn, _) if appConns.contains(appconn) => true + case _ => false + } + } + + override def parse(gatewayContext: GatewayContext): Unit = gatewayContext.getRequest.getRequestURI match { + + case DSSGatewayParser.DSS_URL_REGEX(version, firstName, secondName) => + if (sendResponseWhenNotMatchVersion(gatewayContext, version)) return + var tmpServerName = "dss-" + firstName + "-" + secondName + "-server" + tmpServerName = getServiceNameFromLabel(gatewayContext, tmpServerName) + val serviceName: Option[String] = findCommonService("dss/" + firstName + "/" + secondName, tmpServerName) + if (serviceName.isDefined) { + gatewayContext.getGatewayRoute.setServiceInstance(ServiceInstance(serviceName.get, null)) + } else { + logger.info("Now set default serviceInstance name " + DSSGatewayConfiguration.DSS_SPRING_NAME.getValue + "," + gatewayContext.getRequest.getRequestURI) + gatewayContext.getGatewayRoute.setServiceInstance(ServiceInstance(DSSGatewayConfiguration.DSS_SPRING_NAME.getValue, null)) + } + case DSSGatewayParser.DSS_URL_DEFAULT_REGEX(version, firstName) => + if (sendResponseWhenNotMatchVersion(gatewayContext, version)) return + var tmpServerName = "dss-" + firstName + "-server" + tmpServerName = getServiceNameFromLabel(gatewayContext, tmpServerName) + val serviceName: Option[String] = findCommonService("dss/" + firstName, tmpServerName) + if (serviceName.isDefined) { + gatewayContext.getGatewayRoute.setServiceInstance(ServiceInstance(serviceName.get, null)) + } else { + logger.info("Now set default serviceInstance name " + DSSGatewayConfiguration.DSS_SPRING_NAME.getValue + "," + gatewayContext.getRequest.getRequestURI) + gatewayContext.getGatewayRoute.setServiceInstance(ServiceInstance(DSSGatewayConfiguration.DSS_SPRING_NAME.getValue, null)) + } + case DSSGatewayParser.APPCONN_URL_DEFAULT_REGEX(version, serverName, _) if appConns.contains(serverName) => + if (sendResponseWhenNotMatchVersion(gatewayContext, version)) return + var tmpServerName = serverName + tmpServerName = getServiceNameFromLabel(gatewayContext, tmpServerName) + val serviceName: Option[String] = findCommonService(tmpServerName, tmpServerName) + if (serviceName.isDefined) { + gatewayContext.getGatewayRoute.setServiceInstance(ServiceInstance(serviceName.get, null)) + } else { + logger.info("Now set default serviceInstance name " + DSSGatewayConfiguration.DSS_SPRING_NAME.getValue + "," + gatewayContext.getRequest.getRequestURI) + gatewayContext.getGatewayRoute.setServiceInstance(ServiceInstance(DSSGatewayConfiguration.DSS_SPRING_NAME.getValue, null)) + } + case _ => + } + + private def getServiceNameFromLabel(gatewayContext: GatewayContext, tmpServiceName: String): String = { + var requestUrlLabels = gatewayContext.getRequest.getQueryParams.getOrDefault(DSSGatewayConfiguration.DSS_URL_LABEL_PREFIX.getValue, null) + if(requestUrlLabels == null) requestUrlLabels = gatewayContext.getRequest.getQueryParams.getOrDefault(DSSGatewayConfiguration.DSS_URL_ROUTE_LABEL_PREFIX.getValue,null) + logger.info("Get ServiceName From Label and method is "+ gatewayContext.getRequest.getMethod.toString+",and urlLabels is "+requestUrlLabels) + if (requestUrlLabels == null && gatewayContext.getRequest.getMethod.equalsIgnoreCase("post")) { + val requestBody = Option(gatewayContext.getRequest.getRequestBody) + val routeLabelList = new util.ArrayList[RouteLabel]() + + requestBody match { + //todo form-data resolve + case Some(body) => + val labelBuilderFactory = LabelBuilderFactoryContext.getLabelBuilderFactory + val json = BDPJettyServerHelper.gson.fromJson(body, classOf[java.util.Map[String, Object]]) + val labels: util.List[Label[_]] = json.get(TaskConstant.LABELS) match { + case map: util.Map[String, Object] => labelBuilderFactory.getLabels(map) + case map: util.Map[String, Any] => labelBuilderFactory.getLabels(map.asInstanceOf) + case _ => new util.ArrayList[Label[_]]() + } + labels.filter(label => label.isInstanceOf[RouteLabel]).foreach(label => { + routeLabelList.add(label.asInstanceOf[RouteLabel]) + }) + + case _ => null + } + val labelNameList = routeLabelList.map(routeLabel => routeLabel.getStringValue).toList + if(labelNameList != null && labelNameList.size>0) { + genServiceNameByDSSLabel(labelNameList, tmpServiceName) + }else if(null != requestUrlLabels){ + genServiceNameByDSSLabel(requestUrlLabels.toList, tmpServiceName) + } else tmpServiceName + + } else { + if(requestUrlLabels != null) { + genServiceNameByDSSLabel(requestUrlLabels.toList, tmpServiceName) + }else tmpServiceName + } + } + + private def genServiceNameByDSSLabel(labelList: List[String], tmpServiceName: String): String = { + var resultName = tmpServiceName + if (null != labelList && labelList.size > 0) { + val labelNameList = labelList(0).replace(" ", "").split(",").toList + if (labelNameList.size > 0) { + if (labelNameList.find(name => name.equalsIgnoreCase("dev")).isDefined) { + resultName = tmpServiceName + "-dev" + } else if (labelNameList.find(name => name.equalsIgnoreCase("prod")).isDefined) { + resultName = tmpServiceName + "-prod" + } else if (labelNameList.find(name => name.equalsIgnoreCase("test")).isDefined){ + resultName = tmpServiceName + "-test" + }else{ + resultName = tmpServiceName + } + } + } + resultName + } + + + private def findCommonService(parsedServiceId: String, tmpServerName: String) = findService(parsedServiceId, tmpServerName, services => { + val errorMsg = new TooManyServiceException(s"Cannot find a correct serviceId for parsedServiceId $parsedServiceId, service list is: " + services) + warn("", errorMsg) + throw errorMsg + }) + + protected def findService(parsedServiceId: String, tmpServerName: String, tooManyDeal: List[String] => Option[String]): Option[String] = { + val findIt: (String => Boolean) => Option[String] = op => { + val services = SpringCloudFeignConfigurationCache.getDiscoveryClient + .getServices.filter(op).toList + if (services.length == 1) Some(services.head) + else if (services.length > 1) tooManyDeal(services) + else None + } + //通过匹配到最多的url中的path进行路由,如/dss/framework/workspace/ 会匹配到 dss-framework-workspace-server 而不是 dss-server + // 如果产生了相等的情况，则按照短的service名字为准 比如/dss/getProject, + // 我们可能会匹配到dss-server以及 dss-framework-workspace-server,则选择短名称的dss-server + val findMostCorrect: (String => (String, Int)) => Option[String] = { op => { + val serviceMap = SpringCloudFeignConfigurationCache.getDiscoveryClient + .getServices.map(op).toMap + var count = 0 + var retService: Option[String] = None + serviceMap.foreach { + case (k, v) => if (v > count) { + count = v + retService = Some(k) + } else if (retService.isDefined && v == count && k.length < retService.get.length) { + retService = Some(k) + } + } + retService + } + } + val lowerServiceId = parsedServiceId.toLowerCase + val serverName = tmpServerName.toLowerCase + //findIt(_.toLowerCase == lowerServiceId).orElse(findIt(_.toLowerCase.contains(lowerServiceId))) + findIt(_.toLowerCase == serverName).orElse(findMostCorrect(service => { + (service, lowerServiceId.split("/").count(word => service.contains(word))) + })) + } + + +} + +object DSSGatewayParser { + val DSS_HEADER = normalPath(API_URL_PREFIX) + "rest_[a-zA-Z][a-zA-Z_0-9]*/(v\\d+)/dss/" + val DSS_URL_REGEX = (DSS_HEADER + "([^/]+)/" + "([^/]+)/.+").r + val DSS_URL_DEFAULT_REGEX = (DSS_HEADER + "([^/]+).+").r + + val APPCONN_HEADER = normalPath(API_URL_PREFIX) + "rest_[a-zA-Z][a-zA-Z_0-9]*/(v\\d+)/([^/]+)/" + val APPCONN_URL_DEFAULT_REGEX = (APPCONN_HEADER + "([^/]+).+").r + +} diff --git a/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSRouteLabelParser.scala b/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSRouteLabelParser.scala new file mode 100644 index 000000000..59e4fe40d --- /dev/null +++ b/plugins/linkis/dss-gateway-support/src/main/scala/com/webank/wedatasphere/linkis/gateway/parser/DSSRouteLabelParser.scala @@ -0,0 +1,38 @@ +/* + * Copyright 2019 WeBank + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.webank.wedatasphere.linkis.gateway.parser + +import java.util + +import com.webank.wedatasphere.linkis.gateway.http.GatewayContext +import com.webank.wedatasphere.linkis.gateway.ujes.route.label.RouteLabelParser +import com.webank.wedatasphere.linkis.manager.label.builder.factory.LabelBuilderFactoryContext +import com.webank.wedatasphere.linkis.manager.label.entity.Label +import com.webank.wedatasphere.linkis.manager.label.entity.route.RouteLabel +import com.webank.wedatasphere.linkis.protocol.constants.TaskConstant +import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper +import org.springframework.stereotype.Component + +import scala.collection.JavaConversions._ + + +@Component +class DSSRouteLabelParser extends RouteLabelParser{ + override def parse(gatewayContext: GatewayContext): util.List[RouteLabel] = { + new util.ArrayList[RouteLabel]() + } +} diff --git a/plugins/linkis/linkis-appjoint-entrance/bin/start-linkis-appjoint-entrance.sh b/plugins/linkis/linkis-appjoint-entrance/bin/start-linkis-appjoint-entrance.sh deleted file mode 100644 index 4436def35..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/bin/start-linkis-appjoint-entrance.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/bash -cd `dirname $0` -cd .. -HOME=`pwd` - -export SERVER_PID=$HOME/bin/linkis.pid -export SERVER_LOG_PATH=$HOME/logs -export SERVER_CLASS=com.webank.wedatasphere.linkis.DataWorkCloudApplication - -if test -z "$SERVER_HEAP_SIZE" -then - export SERVER_HEAP_SIZE="512M" -fi - -if test -z "$SERVER_JAVA_OPTS" -then - export SERVER_JAVA_OPTS=" -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$HOME/logs/linkis-gc.log" -fi - -if [[ -f "${SERVER_PID}" ]]; then - pid=$(cat ${SERVER_PID}) - if kill -0 ${pid} >/dev/null 2>&1; then - echo "Server is already running." - exit 1 - fi -fi - -nohup java $SERVER_JAVA_OPTS -cp $HOME/conf:$HOME/lib/* $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/linkis.out & -pid=$! -if [[ -z "${pid}" ]]; then - echo "server $SERVER_NAME start failed!" - exit 1 -else - echo "server $SERVER_NAME start succeeded!" - echo $pid > $SERVER_PID - sleep 1 -fi diff --git a/plugins/linkis/linkis-appjoint-entrance/bin/stop-linkis-appjoint-entrance.sh b/plugins/linkis/linkis-appjoint-entrance/bin/stop-linkis-appjoint-entrance.sh deleted file mode 100644 index a1528811c..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/bin/stop-linkis-appjoint-entrance.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -cd `dirname $0` -cd .. -HOE=`pwd` - -export DSS_ENTRANCE_PID=$HOE/bin/linkis.pid - -function wait_for_DSS_ENGINE_MANAGER_to_die() { - local pid - local count - pid=$1 - timeout=$2 - count=0 - timeoutTime=$(date "+%s") - let "timeoutTime+=$timeout" - currentTime=$(date "+%s") - forceKill=1 - - while [[ $currentTime -lt $timeoutTime ]]; do - $(kill ${pid} > /dev/null 2> /dev/null) - if kill -0 ${pid} > /dev/null 2>&1; then - sleep 3 - else - forceKill=0 - break - fi - currentTime=$(date "+%s") - done - - if [[ forceKill -ne 0 ]]; then - $(kill -9 ${pid} > /dev/null 2> /dev/null) - fi -} - -if [[ ! -f "${DSS_ENTRANCE_PID}" ]]; then - echo "AppJoint Entrance is not running" -else - pid=$(cat ${DSS_ENTRANCE_PID}) - if [[ -z "${pid}" ]]; then - echo "AppJoint Entrance is not running" - else - wait_for_DSS_ENGINE_MANAGER_to_die $pid 40 - $(rm -f ${DSS_ENTRANCE_PID}) - echo "AppJoint Entrance is stopped." - fi -fi diff --git a/plugins/linkis/linkis-appjoint-entrance/conf/application.yml b/plugins/linkis/linkis-appjoint-entrance/conf/application.yml deleted file mode 100644 index 2f21e55c0..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/conf/application.yml +++ /dev/null @@ -1,22 +0,0 @@ -server: - port: 9006 -spring: - application: - name: appjointEntrance - -eureka: - client: - serviceUrl: - defaultZone: http://127.0.0.1:20303/eureka/ - instance: - metadata-map: - test: enjoyyin -management: - endpoints: - web: - exposure: - include: refresh,info -logging: - config: classpath:log4j2.xml -# register-with-eureka: false -# fetch-registry: false diff --git a/plugins/linkis/linkis-appjoint-entrance/conf/linkis.properties b/plugins/linkis/linkis-appjoint-entrance/conf/linkis.properties deleted file mode 100644 index 1317805e2..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/conf/linkis.properties +++ /dev/null @@ -1,38 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - - -wds.linkis.server.restful.scan.packages=com.webank.wedatasphere.linkis.entrance.restful - -wds.linkis.engine.application.name=hiveEngine -wds.linkis.enginemanager.application.name=hiveEngineManager - -wds.linkis.query.application.name=cloud-publicservice - -wds.linkis.console.config.application.name=cloud-publicservice -wds.linkis.engine.creation.wait.time.max=20m -wds.linkis.server.version=v1 -#hadoop config dir -#hadoop.config.dir=/appcom/config/hadoop-config -wds.linkis.entrance.config.logPath=file:///tmp/linkis/ - -wds.linkis.resultSet.store.path=hdfs:///tmp/linkis - -wds.linkis.server.socket.mode=true - - -wds.linkis.entrance.job.persist.wait.max=10s diff --git a/plugins/linkis/linkis-appjoint-entrance/conf/log4j2.xml b/plugins/linkis/linkis-appjoint-entrance/conf/log4j2.xml deleted file mode 100644 index f48149dd2..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/conf/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/plugins/linkis/linkis-appjoint-entrance/pom.xml b/plugins/linkis/linkis-appjoint-entrance/pom.xml deleted file mode 100644 index bb7db1901..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/pom.xml +++ /dev/null @@ -1,141 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - linkis-appjoint-entrance - ${dss.version} - - - - com.webank.wedatasphere.dss - dss-appjoint-core - ${dss.version} - - - - com.webank.wedatasphere.dss - dss-appjoint-loader - ${dss.version} - - - - - com.webank.wedatasphere.linkis - linkis-ujes-entrance - ${linkis.version} - - - org.springframework.cloud - spring-cloud-starter-openfeign - - - - - com.webank.wedatasphere.linkis - linkis-cloudRPC - ${linkis.version} - - - com.webank.wedatasphere.linkis - linkis-storage - ${linkis.version} - - - com.webank.wedatasphere.linkis - linkis-httpclient - ${linkis.version} - - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - linkis-appjoint-entrance - false - false - - src/main/assembly/distribution.xml - - - - - - - src/main/java - - **/*.xml - - - - src/main/resources - - **/*.properties - **/application.yml - **/bootstrap.yml - **/log4j2.xml - - - - - - - \ No newline at end of file diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/assembly/distribution.xml b/plugins/linkis/linkis-appjoint-entrance/src/main/assembly/distribution.xml deleted file mode 100644 index b3ac1a0b5..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/assembly/distribution.xml +++ /dev/null @@ -1,335 +0,0 @@ - - - - linkis-appjoint-entrance - - zip - - true - linkis-appjoint-entrance - - - - - - lib - true - true - false - true - true - - - antlr:antlr:jar - aopalliance:aopalliance:jar - asm:asm:jar - cglib:cglib:jar - - - - - - - com.fasterxml.jackson.core:jackson-annotations:jar - com.fasterxml.jackson.core:jackson-core:jar - com.fasterxml.jackson.core:jackson-databind:jar - - com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar - com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar - com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar - com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar - com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar - com.fasterxml.jackson.module:jackson-module-parameter-names:jar - com.fasterxml.jackson.module:jackson-module-paranamer:jar - com.fasterxml.jackson.module:jackson-module-scala_2.11:jar - - - com.google.code.findbugs:annotations:jar - - com.google.code.gson:gson:jar - com.google.guava:guava:jar - com.google.inject:guice:jar - - com.netflix.archaius:archaius-core:jar - com.netflix.eureka:eureka-client:jar - com.netflix.eureka:eureka-core:jar - com.netflix.hystrix:hystrix-core:jar - com.netflix.netflix-commons:netflix-commons-util:jar - com.netflix.netflix-commons:netflix-eventbus:jar - com.netflix.netflix-commons:netflix-infix:jar - com.netflix.netflix-commons:netflix-statistics:jar - com.netflix.ribbon:ribbon:jar - com.netflix.ribbon:ribbon-core:jar - com.netflix.ribbon:ribbon-eureka:jar - com.netflix.ribbon:ribbon-httpclient:jar - com.netflix.ribbon:ribbon-loadbalancer:jar - com.netflix.ribbon:ribbon-transport:jar - com.netflix.servo:servo-core:jar - com.sun.jersey.contribs:jersey-apache-client4:jar - com.sun.jersey:jersey-client:jar - com.sun.jersey:jersey-core:jar - com.sun.jersey:jersey-json:jar - com.sun.jersey:jersey-server:jar - com.sun.jersey:jersey-servlet:jar - com.sun.xml.bind:jaxb-impl:jar - com.thoughtworks.paranamer:paranamer:jar - com.thoughtworks.xstream:xstream:jar - com.webank.wedatasphere.linkis:linkis-common:jar - com.webank.wedatasphere.linkis:linkis-module:jar - - - - - - - - - - commons-httpclient:commons-httpclient:jar - commons-io:commons-io:jar - commons-jxpath:commons-jxpath:jar - commons-lang:commons-lang:jar - commons-logging:commons-logging:jar - commons-net:commons-net:jar - commons-pool:commons-pool:jar - io.micrometer:micrometer-core:jar - io.netty:netty:jar - io.netty:netty-all:jar - io.netty:netty-buffer:jar - io.netty:netty-codec:jar - io.netty:netty-codec-http:jar - io.netty:netty-common:jar - io.netty:netty-handler:jar - io.netty:netty-transport:jar - io.netty:netty-transport-native-epoll:jar - io.reactivex:rxjava:jar - io.reactivex:rxnetty:jar - io.reactivex:rxnetty-contexts:jar - io.reactivex:rxnetty-servo:jar - javax.activation:activation:jar - javax.annotation:javax.annotation-api:jar - javax.inject:javax.inject:jar - javax.servlet:javax.servlet-api:jar - javax.servlet.jsp:jsp-api:jar - javax.validation:validation-api:jar - javax.websocket:javax.websocket-api:jar - javax.ws.rs:javax.ws.rs-api:jar - javax.xml.bind:jaxb-api:jar - javax.xml.stream:stax-api:jar - joda-time:joda-time:jar - log4j:log4j:jar - mysql:mysql-connector-java:jar - org.antlr:antlr-runtime:jar - org.antlr:stringtemplate:jar - - org.apache.commons:commons-math:jar - org.apache.commons:commons-math3:jar - org.apache.curator:curator-client:jar - org.apache.curator:curator-framework:jar - org.apache.curator:curator-recipes:jar - org.apache.directory.api:api-asn1-api:jar - org.apache.directory.api:api-util:jar - org.apache.directory.server:apacheds-i18n:jar - org.apache.directory.server:apacheds-kerberos-codec:jar - - - - - - - - org.apache.logging.log4j:log4j-api:jar - org.apache.logging.log4j:log4j-core:jar - org.apache.logging.log4j:log4j-jul:jar - org.apache.logging.log4j:log4j-slf4j-impl:jar - org.apache.zookeeper:zookeeper:jar - org.aspectj:aspectjweaver:jar - org.bouncycastle:bcpkix-jdk15on:jar - org.bouncycastle:bcprov-jdk15on:jar - org.codehaus.jackson:jackson-jaxrs:jar - org.codehaus.jackson:jackson-xc:jar - org.codehaus.jettison:jettison:jar - org.codehaus.woodstox:stax2-api:jar - org.codehaus.woodstox:woodstox-core-asl:jar - org.eclipse.jetty:jetty-annotations:jar - org.eclipse.jetty:jetty-client:jar - org.eclipse.jetty:jetty-continuation:jar - org.eclipse.jetty:jetty-http:jar - org.eclipse.jetty:jetty-io:jar - org.eclipse.jetty:jetty-plus:jar - org.eclipse.jetty:jetty-security:jar - org.eclipse.jetty:jetty-server:jar - org.eclipse.jetty:jetty-servlet:jar - org.eclipse.jetty:jetty-servlets:jar - org.eclipse.jetty:jetty-util:jar - org.eclipse.jetty:jetty-webapp:jar - org.eclipse.jetty:jetty-xml:jar - org.eclipse.jetty.websocket:javax-websocket-client-impl:jar - org.eclipse.jetty.websocket:javax-websocket-server-impl:jar - org.eclipse.jetty.websocket:websocket-api:jar - org.eclipse.jetty.websocket:websocket-client:jar - org.eclipse.jetty.websocket:websocket-common:jar - org.eclipse.jetty.websocket:websocket-server:jar - org.eclipse.jetty.websocket:websocket-servlet:jar - org.fusesource.leveldbjni:leveldbjni-all:jar - org.glassfish.hk2:class-model:jar - org.glassfish.hk2:config-types:jar - org.glassfish.hk2.external:aopalliance-repackaged:jar - org.glassfish.hk2.external:asm-all-repackaged:jar - org.glassfish.hk2.external:bean-validator:jar - org.glassfish.hk2.external:javax.inject:jar - org.glassfish.hk2:hk2:jar - org.glassfish.hk2:hk2-api:jar - org.glassfish.hk2:hk2-config:jar - org.glassfish.hk2:hk2-core:jar - org.glassfish.hk2:hk2-locator:jar - org.glassfish.hk2:hk2-runlevel:jar - org.glassfish.hk2:hk2-utils:jar - org.glassfish.hk2:osgi-resource-locator:jar - org.glassfish.hk2:spring-bridge:jar - org.glassfish.jersey.bundles:jaxrs-ri:jar - org.glassfish.jersey.bundles.repackaged:jersey-guava:jar - org.glassfish.jersey.containers:jersey-container-servlet:jar - org.glassfish.jersey.containers:jersey-container-servlet-core:jar - org.glassfish.jersey.core:jersey-client:jar - org.glassfish.jersey.core:jersey-common:jar - org.glassfish.jersey.core:jersey-server:jar - org.glassfish.jersey.ext:jersey-entity-filtering:jar - org.glassfish.jersey.ext:jersey-spring3:jar - org.glassfish.jersey.media:jersey-media-jaxb:jar - org.glassfish.jersey.media:jersey-media-json-jackson:jar - org.glassfish.jersey.media:jersey-media-multipart:jar - org.hdrhistogram:HdrHistogram:jar - org.javassist:javassist:jar - org.json4s:json4s-ast_2.11:jar - org.json4s:json4s-core_2.11:jar - org.json4s:json4s-jackson_2.11:jar - org.jvnet.mimepull:mimepull:jar - org.jvnet:tiger-types:jar - org.latencyutils:LatencyUtils:jar - org.mortbay.jasper:apache-el:jar - org.mortbay.jetty:jetty:jar - org.mortbay.jetty:jetty-util:jar - org.ow2.asm:asm-analysis:jar - org.ow2.asm:asm-commons:jar - org.ow2.asm:asm-tree:jar - org.reflections:reflections:jar - org.scala-lang.modules:scala-parser-combinators_2.11:jar - org.scala-lang.modules:scala-xml_2.11:jar - org.scala-lang:scala-compiler:jar - org.scala-lang:scala-library:jar - org.scala-lang:scala-reflect:jar - org.scala-lang:scalap:jar - org.slf4j:jul-to-slf4j:jar - org.slf4j:slf4j-api:jar - org.springframework.boot:spring-boot:jar - org.springframework.boot:spring-boot-actuator:jar - org.springframework.boot:spring-boot-actuator-autoconfigure:jar - org.springframework.boot:spring-boot-autoconfigure:jar - org.springframework.boot:spring-boot-starter:jar - org.springframework.boot:spring-boot-starter-actuator:jar - org.springframework.boot:spring-boot-starter-aop:jar - org.springframework.boot:spring-boot-starter-jetty:jar - org.springframework.boot:spring-boot-starter-json:jar - org.springframework.boot:spring-boot-starter-log4j2:jar - org.springframework.boot:spring-boot-starter-web:jar - org.springframework.cloud:spring-cloud-commons:jar - org.springframework.cloud:spring-cloud-config-client:jar - org.springframework.cloud:spring-cloud-context:jar - - org.springframework.cloud:spring-cloud-netflix-core:jar - org.springframework.cloud:spring-cloud-netflix-eureka-client:jar - - org.springframework.cloud:spring-cloud-starter:jar - org.springframework.cloud:spring-cloud-starter-config:jar - org.springframework.cloud:spring-cloud-starter-eureka:jar - org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar - org.springframework.cloud:spring-cloud-starter-netflix-eureka-client:jar - org.springframework.cloud:spring-cloud-starter-netflix-ribbon:jar - org.springframework.security:spring-security-crypto:jar - org.springframework.security:spring-security-rsa:jar - org.springframework:spring-aop:jar - org.springframework:spring-beans:jar - org.springframework:spring-context:jar - org.springframework:spring-core:jar - org.springframework:spring-expression:jar - org.springframework:spring-jcl:jar - org.springframework:spring-web:jar - org.springframework:spring-webmvc:jar - org.tukaani:xz:jar - org.yaml:snakeyaml:jar - - xerces:xercesImpl:jar - xmlenc:xmlenc:jar - xmlpull:xmlpull:jar - xpp3:xpp3_min:jar - - - - - - - - ${basedir}/conf - - * - - 0777 - conf - unix - - - ${basedir}/bin - - * - - 0777 - bin - unix - - - . - - */** - - logs - - - - - diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/java/com/webank/wedatasphere/dss/linkis/appjoint/entrance/AppJointEntranceRestfulApi.java b/plugins/linkis/linkis-appjoint-entrance/src/main/java/com/webank/wedatasphere/dss/linkis/appjoint/entrance/AppJointEntranceRestfulApi.java deleted file mode 100644 index 2b12ad7cb..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/java/com/webank/wedatasphere/dss/linkis/appjoint/entrance/AppJointEntranceRestfulApi.java +++ /dev/null @@ -1,96 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution; -import com.webank.wedatasphere.dss.appjoint.execution.core.CallbackLongTermNodeExecution; -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.appjoint.AppJointManager; -import com.webank.wedatasphere.linkis.entrance.EntranceContext; -import com.webank.wedatasphere.linkis.server.Message; -import org.apache.commons.lang.StringUtils; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Component; - -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; -import javax.ws.rs.Consumes; -import javax.ws.rs.POST; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; -import java.util.Map; - -/** - * created by cooperyang on 2019/9/30 - * Description: - */ -@Path("/entrance") -@Component -@Produces(MediaType.APPLICATION_JSON) -@Consumes(MediaType.APPLICATION_JSON) -public class AppJointEntranceRestfulApi { - - public static final String APPJOINT_NAME_STR = "appJointName"; - - public static final String APPJOINT_LOG_STR = "log"; - - public static final String APPJOINT_STATUS_STR = "status"; - - public static final String APPJOINT_PROGRESS_STR = "progress"; - - public static final String ENGINE_TAG_STR = "engineTag"; - - public static final String CALL_BACK_URL = "/api/entrance/callback"; - - /** - * callback接口是为了接收外部系统进行回馈各种信息的接口，外部系统调用此接口必须提供的参数有entranceEngine的唯一标识， - * 因为只有entranceEngine的唯一标识，才能进行定位到相应的job - * jsonNode 是外部系统传递的参数,这些参数可以包含有日志，进度 和 状态等等 - */ - @POST - @Path("/callback") - public Response callback(@Context HttpServletRequest request, - @Context HttpServletResponse response, - Map params) { - Object appJointName = params.get(APPJOINT_NAME_STR); - if(appJointName == null || StringUtils.isBlank(appJointName.toString())) { - return error(APPJOINT_NAME_STR + "为空！"); - } - AppJoint appJoint = AppJointManager.getAppJoint(appJointName.toString()); - if(appJoint == null) { - return error("找不到AppJoint: " + appJointName); - } - NodeExecution nodeExecution = appJoint.getNodeExecution(); - if(!(nodeExecution instanceof CallbackLongTermNodeExecution)) { - return error("找不到CallbackLongTermNodeExecution来处理该请求，实际NodeExecution为 " + nodeExecution.getClass().getSimpleName()); - } - ((CallbackLongTermNodeExecution) nodeExecution).acceptCallback(params); - Message message = Message.ok("处理成功!"); - message.setMethod(CALL_BACK_URL); - return Message.messageToResponse(message); - } - - private Response error(String msg) { - Message message = Message.error(msg); - message.setMethod(CALL_BACK_URL); - return Message.messageToResponse(message); - } -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/appjoint/AppJointManager.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/appjoint/AppJointManager.scala deleted file mode 100644 index 5a2dea709..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/appjoint/AppJointManager.scala +++ /dev/null @@ -1,75 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.appjoint - -import java.util -import java.util.concurrent.TimeUnit - -import com.google.common.cache.CacheBuilder -import com.webank.wedatasphere.dss.appjoint.AppJoint -import com.webank.wedatasphere.dss.appjoint.execution.scheduler.ListenerEventBusNodeExecutionScheduler -import com.webank.wedatasphere.dss.appjoint.loader.AppJointLoader -import com.webank.wedatasphere.dss.common.protocol.RequestDSSApplication -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf.AppJointEntranceConfiguration -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import com.webank.wedatasphere.linkis.rpc.Sender - -/** - * Created by enjoyyin on 2019/11/13. - */ -object AppJointManager extends Logging{ - - private val scheduler = new ListenerEventBusNodeExecutionScheduler - - scheduler.start() - - - private val sender = Sender.getSender(AppJointEntranceConfiguration.DSS_SERVER_SPRING_APPLICATION.getValue) - - private val urlStr:String = "url" - - def getScheduler:ListenerEventBusNodeExecutionScheduler = scheduler - - - /** - * 通过appJointName获取一个可以使用的AppJoint - * @param appJointName appjoint的名字,例如 sendemail,qualitis - * @return - */ - - - def getAppJoint(appJointName: String): AppJoint = { - val request = RequestDSSApplication(appJointName) - val result = Utils.tryCatch{ - sender.ask(request) - }{ - t:Throwable => logger.warn(s"can not get application $appJointName from dss-Server", t) - null - } - var params:java.util.Map[String, Object] = new util.HashMap[String, Object]() - val baseUrl:String = if (result != null){ - result match { - case map:java.util.Map[String, Object] => params = map - if (null != map.get(urlStr)) map.get(urlStr).toString else "" - case _ => "" - } - }else "" - AppJointLoader.getAppJointLoader.getAppJoint("", appJointName, params) - } - -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointConst.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointConst.scala deleted file mode 100644 index 6aa98b580..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointConst.scala +++ /dev/null @@ -1,39 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf - -import java.lang -import java.lang.reflect.Type - -import com.google.gson._ -import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper - -/** - * created by cooperyang on 2019/11/11 - * Description: - */ -object AppJointConst { - - implicit val gson = new GsonBuilder().setDateFormat("yyyy-MM-dd'T'HH:mm:ssZ").serializeNulls - .registerTypeAdapter(classOf[java.lang.Double], new JsonSerializer[java.lang.Double] { - override def serialize(t: lang.Double, `type`: Type, jsonSerializationContext: JsonSerializationContext): JsonElement = - if(t == t.longValue()) new JsonPrimitive(t.longValue()) else new JsonPrimitive(t) - }).create - - val WORKFLOW_SHARED_NODES_JOBIDS:String = "workflow.shared.nodes.jobids" -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointEntranceConfiguration.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointEntranceConfiguration.scala deleted file mode 100644 index 9bb45ed39..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointEntranceConfiguration.scala +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf - -import com.webank.wedatasphere.linkis.common.conf.CommonVars - -/** - * created by cooperyang on 2019/9/26 - * Description: - */ -object AppJointEntranceConfiguration { - val APP_JOINT_CLASS = CommonVars("wds.linkis.external.appjoint", - "com.webank.wedatasphere.dss.linkis.appjoint.example.ExampleAppJoint") - - val APPLICATION_SPRING_APPLICATION = CommonVars("wds.linkis.application.spring.name", "cloud-application") - - val CALLBACK_TIME_OUT:CommonVars[Long] = CommonVars[Long]("wds.linkis.callback.timeout", 5 * 60 * 1000L) - - - val PUBLIC_SERVICE_SPRING_APPLICATION = CommonVars("wds.linkis.publicservice.spring.name", "cloud-publicserivce") - - val GATEWAY_SPRING_APPLICATION = CommonVars("wds.linkis.gateway.spring.name", "dataworkcloud-gateway") - - val DSS_SERVER_SPRING_APPLICATION = CommonVars("wds.linkis.dssserver.spring.name", "dss-server") -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointEntranceSpringConfiguration.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointEntranceSpringConfiguration.scala deleted file mode 100644 index 7c1d35edd..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/conf/AppJointEntranceSpringConfiguration.scala +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf - -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.execute.AppJointEngineExecutorManagerImpl -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.parser.AppJointEntranceParser -import com.webank.wedatasphere.linkis.entrance.EntranceParser -import com.webank.wedatasphere.linkis.entrance.annotation._ -import com.webank.wedatasphere.linkis.entrance.execute._ -import com.webank.wedatasphere.linkis.scheduler.queue.GroupFactory -import org.slf4j.LoggerFactory -import org.springframework.beans.factory.annotation.Autowired -import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean -import org.springframework.context.annotation.Configuration - -/** - * created by chaogefeng on 2019/9/26 17:44 - * Description: - */ -@Configuration -class AppJointEntranceSpringConfiguration { - private val logger = LoggerFactory.getLogger(classOf[AppJointEntranceSpringConfiguration]) - @EntranceExecutorManagerBeanAnnotation - def generateExternalEntranceExecutorManager(@GroupFactoryBeanAnnotation.GroupFactoryAutowiredAnnotation groupFactory: GroupFactory, - @EngineBuilderBeanAnnotation.EngineBuilderAutowiredAnnotation engineBuilder: EngineBuilder, - @EngineRequesterBeanAnnotation.EngineRequesterAutowiredAnnotation engineRequester: EngineRequester, - @EngineSelectorBeanAnnotation.EngineSelectorAutowiredAnnotation engineSelector: EngineSelector, - @EngineManagerBeanAnnotation.EngineManagerAutowiredAnnotation engineManager: EngineManager, - @Autowired entranceExecutorRulers: Array[EntranceExecutorRuler]): EntranceExecutorManager = - new AppJointEngineExecutorManagerImpl(groupFactory, engineBuilder, engineRequester, engineSelector, engineManager, entranceExecutorRulers) - - - @EntranceParserBeanAnnotation - def generateEntranceParser():EntranceParser = { - new AppJointEntranceParser() - } - -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/AppJointEntranceIllegalParamException.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/AppJointEntranceIllegalParamException.scala deleted file mode 100644 index cdb6ca9be..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/AppJointEntranceIllegalParamException.scala +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.exception - -import com.webank.wedatasphere.linkis.common.exception.ErrorException - -/** - * created by cooperyang on 2019/10/8 - * Description: - */ -case class AppJointEntranceIllegalParamException(errorMsg:String) extends ErrorException(70012, errorMsg) - -case class AppJointErrorException(errorMsg:String) extends ErrorException(70058, errorMsg) \ No newline at end of file diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/OperateNotSupportedException.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/OperateNotSupportedException.scala deleted file mode 100644 index 4177aaf9b..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/OperateNotSupportedException.scala +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.exception - -import com.webank.wedatasphere.linkis.common.exception.ErrorException - -/** - * created by cooperyang on 2019/9/30 - * Description: - */ -case class OperateNotSupportedException(errMsg:String) extends ErrorException(81235, errMsg) diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/RPCFailedException.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/RPCFailedException.scala deleted file mode 100644 index 0c4796ca0..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/exception/RPCFailedException.scala +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.exception - -import com.webank.wedatasphere.linkis.common.exception.ErrorException - -/** - * created by cooperyang on 2019/10/12 - * Description: - */ -case class RPCFailedException(errMsg:String) extends ErrorException(80056, errMsg) - diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointEngineExecutorManagerImpl.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointEngineExecutorManagerImpl.scala deleted file mode 100644 index 709c19990..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointEngineExecutorManagerImpl.scala +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.execute - -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.job.AppJointEntranceJob -import com.webank.wedatasphere.linkis.common.utils.Logging -import com.webank.wedatasphere.linkis.entrance.execute._ -import com.webank.wedatasphere.linkis.entrance.execute.impl.EntranceExecutorManagerImpl -import com.webank.wedatasphere.linkis.scheduler.executer.Executor -import com.webank.wedatasphere.linkis.scheduler.listener.ExecutorListener -import com.webank.wedatasphere.linkis.scheduler.queue.{GroupFactory, Job, SchedulerEvent} - -import scala.concurrent.duration.Duration - -class AppJointEngineExecutorManagerImpl(groupFactory: GroupFactory, - engineBuilder: EngineBuilder, - engineRequester: EngineRequester, - engineSelector: EngineSelector, - engineManager: EngineManager, - entranceExecutorRulers: Array[EntranceExecutorRuler]) - extends EntranceExecutorManagerImpl(groupFactory, engineBuilder, engineRequester, - engineSelector, engineManager, entranceExecutorRulers) with Logging { - override def setExecutorListener(executorListener: ExecutorListener): Unit = super.setExecutorListener(executorListener) - - override def initialEntranceEngine(engine: EntranceEngine): Unit = super.initialEntranceEngine(engine) - - protected override def createExecutor(schedulerEvent: SchedulerEvent): EntranceEngine = schedulerEvent match { - case job: AppJointEntranceJob => - val externalEngineParams = job.getParams - val engine = new AppJointEntranceEngine(externalEngineParams) - engine - case _ => null - } - - override protected def findExecutors(job: Job): Array[EntranceEngine] = super.findExecutors(job) - - override def askExecutor(schedulerEvent: SchedulerEvent): Option[Executor] = schedulerEvent match { - case job: AppJointEntranceJob => - Some(createExecutor(schedulerEvent)) - } - - override def askExecutor(schedulerEvent: SchedulerEvent, wait: Duration): Option[Executor] = schedulerEvent match { - case job: AppJointEntranceJob => - Some(createExecutor(schedulerEvent)) - } - - - override def getById(id: Long): Option[Executor] = super.getById(id) - - override def getByGroup(groupName: String): Array[Executor] = super.getByGroup(groupName) - - protected override def delete(executor: Executor): Unit = super.delete(executor) - - override def shutdown(): Unit = super.shutdown() -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointEntranceEngine.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointEntranceEngine.scala deleted file mode 100644 index e16c9b93e..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointEntranceEngine.scala +++ /dev/null @@ -1,228 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.execute - -import java.util -import java.util.UUID - -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution -import com.webank.wedatasphere.dss.appjoint.execution.async.NodeExecutionResponseListener -import com.webank.wedatasphere.dss.appjoint.execution.common._ -import com.webank.wedatasphere.dss.appjoint.execution.core._ -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.appjoint.AppJointManager -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf.AppJointConst -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.exception.OperateNotSupportedException -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.job.AppJointExecuteRequest -import com.webank.wedatasphere.linkis.common.exception.ErrorException -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import com.webank.wedatasphere.linkis.entrance.execute.{EngineExecuteAsynReturn, EntranceEngine, EntranceJob} -import com.webank.wedatasphere.linkis.entrance.interceptor.impl.CustomVariableUtils -import com.webank.wedatasphere.linkis.protocol.engine.{JobProgressInfo, RequestTask} -import com.webank.wedatasphere.linkis.protocol.query.RequestPersistTask -import com.webank.wedatasphere.linkis.scheduler.executer._ -import com.webank.wedatasphere.linkis.scheduler.queue.Job -import org.apache.commons.lang.StringUtils - -import scala.collection.mutable.ArrayBuffer -import scala.collection.JavaConversions._ -/** - * created by chaogefeng on 2019/9/26 20:23 - * Description: - */ -class AppJointEntranceEngine(properties: util.Map[String, Any]) - extends EntranceEngine(id = 0) with SingleTaskOperateSupport with SingleTaskInfoSupport { - - private val defaultProgress = 0.0f - private val defaultLog = "" - - - private var nodeExecution:NodeExecution = _ - - private var appJointAction:NodeExecutionAction = _ - - - private var response:AppJointEntranceAsyncExecuteResponse = _ - - private var job:Job = _ - - - def setJob(job: Job):Unit = this.job = job - def getJob:Job = this.job - - def setResponse(response:AppJointEntranceAsyncExecuteResponse):Unit = this.response = response - def getResponse:AppJointEntranceAsyncExecuteResponse = this.response - - - - /** - * 设置一个tag是用一个唯一记号标识一个AppJointEntranceEngine，由于每一个job都会对应一个AppJointEntranceEngine - * 所以在callback的时候带上这个 - */ - private val tag = UUID.randomUUID().toString - - override protected def callExecute(request: RequestTask): EngineExecuteAsynReturn = null - - def getTag:String = this.tag - - override def kill(): Boolean = { - nodeExecution match { - case killable:Killable => if (appJointAction != null) killable.kill(appJointAction) else false - case _ => logger.warn(s"${nodeExecution.getClass.getSimpleName} cannot be killed") - throw OperateNotSupportedException(s"${nodeExecution.getClass.getSimpleName} cannot be killed") - } - } - - override def pause(): Boolean = { - throw OperateNotSupportedException(s"${nodeExecution.getClass.getSimpleName} cannot be paused") - } - - override def resume(): Boolean = { - throw OperateNotSupportedException(s"${nodeExecution.getClass.getSimpleName} cannot be resumed") - } - - - override def progress(): Float = { - nodeExecution match { - case procedure:Procedure => if (appJointAction != null) procedure.progress(appJointAction) else defaultProgress - case _ => logger.debug(s"${nodeExecution.getClass.getSimpleName} cannot get progress") - defaultProgress - } - } - - override def getProgressInfo: Array[JobProgressInfo] = Array.empty - - override def log(): String = { - nodeExecution match { - case procedure:Procedure => if (appJointAction != null) procedure.log(appJointAction) else defaultLog - case _ => logger.debug(s"${nodeExecution.getClass.getSimpleName} cannot get log") - defaultLog - } - } - - def generateContext(node: AppJointNode, runTimeMap:util.Map[String, Object]):NodeContext = { - node match { - case commonNode:CommonAppJointNode => val nodeContext = new AppJointNodeContextImpl() - Utils.tryCatch{ - nodeContext.setAppJointNode(node) - nodeContext.setJob(getJob) - nodeContext.setRuntimeMap(runTimeMap) - nodeContext.setStorePath(job.asInstanceOf[EntranceJob].getTask.asInstanceOf[RequestPersistTask].getResultLocation) - nodeContext.setUser(runTimeMap.get("user").asInstanceOf[String]) - nodeContext.setStorePath(runTimeMap.get("storePath").asInstanceOf[String]) - if (runTimeMap.containsKey(AppJointConst.WORKFLOW_SHARED_NODES_JOBIDS)){ - nodeContext.setIsReadNode(true) - runTimeMap.get(AppJointConst.WORKFLOW_SHARED_NODES_JOBIDS) match { - case ids:util.Map[String, String] => val jobIds = new ArrayBuffer[Long]() - ids foreach { - case (k ,v) => Utils.tryAndWarnMsg(jobIds += v.toLong)(s"$v is not a long type") - } - nodeContext.setJobIdsOfShareNode(jobIds.toArray) - case _ => - } - }else{ - nodeContext.setIsReadNode(false) - } - }{ - case e:Exception => logger.warn("创建nodeContext时出现异常,警告信息", e) - } - nodeContext - case _ => logger.warn(s"$node is not a CommonAppJointNode") - null - } - } - - - override def close(): Unit = {} - - override def execute(executeRequest: ExecuteRequest): ExecuteResponse = executeRequest match { - case appJointExecuteRequest:AppJointExecuteRequest => - val node = appJointExecuteRequest.node - val runTimeMap = appJointExecuteRequest.runTimeParams - val nodeContext = generateContext(node, runTimeMap) - val nodeType = nodeContext.getAppJointNode.getNodeType - val realAppJointType = if (nodeType.contains(".")) nodeType.substring(0, nodeType.indexOf(".")) else nodeType - val appJoint = AppJointManager.getAppJoint(realAppJointType) - if((realAppJointType.toLowerCase()).contains("datacheck")){ - replaceCustomVariables(nodeContext.getRuntimeMap) - } - val user = if (null != runTimeMap.get("user")) runTimeMap.get("user").toString else null - val session = if (StringUtils.isNotEmpty(user)){ - if (appJoint.getSecurityService != null) appJoint.getSecurityService.login(user) else null - } else null - this.nodeExecution = appJoint.getNodeExecution - this.nodeExecution match{ - case longTermNodeExecution:LongTermNodeExecution => longTermNodeExecution.setScheduler(AppJointManager.getScheduler) - case _ => - } - if (!nodeExecution.canExecute(node, nodeContext, session)) throw new ErrorException(70058, s"appJoint ${appJoint.getAppJointName} can not execute $appJointExecuteRequest.") - this.nodeExecution.execute(node, nodeContext, session) match { - case completedResponse:CompletedNodeExecutionResponse => - if (completedResponse.isSucceed) SuccessExecuteResponse() - else { - logger.error(s"Failed Reason is ${completedResponse.getErrorMsg}", completedResponse.getException) - ErrorExecuteResponse(completedResponse.getErrorMsg, completedResponse.getException) - } - case asyncAppJointResponse:AsyncNodeExecutionResponse => - appJointAction = asyncAppJointResponse.getAction - response = new AppJointEntranceAsyncExecuteResponse - asyncAppJointResponse.addListener(new NodeExecutionResponseListener{ - override def onNodeExecutionCompleted(resp: CompletedNodeExecutionResponse): Unit = response.notifyStatus(resp) - }) - response - case _ => - logger.warn(s"Can not solve this type of response $response") - ErrorExecuteResponse(s"Can not solve this type of response $response", - new ErrorException(80056, s"Can not solve this type of response $response")) - } - case _ => - ErrorExecuteResponse(s"cannot do this executeRequest $executeRequest", - new ErrorException(80056, s"cannot do this executeRequest $executeRequest")) - } - private def replaceCustomVariables(runTimeMap:java.util.Map[String, Object]):Unit = { - val key = "check.object" - val value:String = if (null != runTimeMap.get(key)) runTimeMap.get(key).toString else "" - val task = new RequestPersistTask - task.setExecutionCode(value) - task.setParams(new util.HashMap[String, Object]()) - val (result, code) = CustomVariableUtils.replaceCustomVar(task, "sql") - logger.info(s"after code replace code is $code") - if (result) runTimeMap(key) = code - } -} - - -case class AppJointEntranceExecuteException(errMsg:String) extends ErrorException(70046, errMsg) - -class AppJointEntranceAsyncExecuteResponse extends AsynReturnExecuteResponse with Logging{ - - var rs:ExecuteResponse => Unit = _ - - def notifyStatus(response: CompletedNodeExecutionResponse):Unit = { - val executeResponse = if(response.isSucceed) SuccessExecuteResponse() - else ErrorExecuteResponse(response.getErrorMsg, response.getException) - if (rs == null) this synchronized(while(rs == null) this.wait(1000)) - rs(executeResponse) - } - - override def notify(rs: ExecuteResponse => Unit): Unit = { - this.rs = rs - this synchronized notifyAll() - } -} - - diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointNodeContextImpl.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointNodeContextImpl.scala deleted file mode 100644 index 81657bbbe..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/execute/AppJointNodeContextImpl.scala +++ /dev/null @@ -1,239 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.execute - -import java.util -import java.util.concurrent.atomic.AtomicInteger - -import com.google.gson.Gson -import com.webank.wedatasphere.dss.appjoint.execution.core.NodeContextImpl -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf.AppJointEntranceConfiguration -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.exception.{AppJointErrorException, RPCFailedException} -import com.webank.wedatasphere.linkis.DataWorkCloudApplication -import com.webank.wedatasphere.linkis.common.exception.ErrorException -import com.webank.wedatasphere.linkis.common.io.resultset.{ResultSet, ResultSetReader, ResultSetWriter} -import com.webank.wedatasphere.linkis.common.io.{FsPath, MetaData, Record} -import com.webank.wedatasphere.linkis.common.utils.{ByteTimeUtils, Logging, Utils} -import com.webank.wedatasphere.linkis.entrance.EntranceContext -import com.webank.wedatasphere.linkis.protocol.query.{RequestPersistTask, RequestQueryTask, ResponsePersist} -import com.webank.wedatasphere.linkis.rpc.Sender -import com.webank.wedatasphere.linkis.scheduler.queue.Job -import com.webank.wedatasphere.linkis.storage.resultset.table.{TableMetaData, TableRecord} -import com.webank.wedatasphere.linkis.storage.resultset.{ResultSetFactory, ResultSetReader, ResultSetWriter, StorageResultSetWriter} -import com.webank.wedatasphere.linkis.storage.{FSFactory, LineMetaData, LineRecord} -import org.apache.commons.lang.StringUtils - -import scala.collection.JavaConversions._ -import scala.collection.mutable.ArrayBuffer -/** - * created by cooperyang on 2019/10/12 - * Description: - */ -class AppJointNodeContextImpl extends NodeContextImpl with Logging{ - - - //querySender 是用来进行访query模块 - val publicServiceSender:Sender = Sender.getSender(AppJointEntranceConfiguration.PUBLIC_SERVICE_SPRING_APPLICATION.getValue) - - val entranceContext:EntranceContext = DataWorkCloudApplication.getApplicationContext.getBean(classOf[EntranceContext]) - - var job:Job = _ - - var storePath:String = _ - - val SUCCESS_FLAG = 0 - - val TASK_MAP_KEY = "task" - - def setJob(job:Job):Unit = this.job = job - - private val resultSetFactory = ResultSetFactory.getInstance - - private val aliasNum = new AtomicInteger(0) - - private val defaultResultSetAlias = "" - - /** - * 通过jobId拿回一个task - * @param jobId - * @return - */ - - override def getJobById(jobId: Long): RequestPersistTask = { - - val requestQueryTask = new RequestQueryTask() - requestQueryTask.setTaskID(jobId) - requestQueryTask.setSource(null) - val task = Utils.tryCatch{ - val taskResponse = publicServiceSender.ask(requestQueryTask) - taskResponse match { - case responsePersist:ResponsePersist => val status = responsePersist.getStatus - if (status != SUCCESS_FLAG){ - logger.error(s"query from jobHistory status failed, status is $status") - throw AppJointErrorException("query from jobHistory status failed") - }else{ - val data = responsePersist.getData - data.get(TASK_MAP_KEY) match { - case tasks:util.List[util.Map[String, Object]] => tasks.get(0) match { - case map:util.Map[String, Object] => val gson = new Gson() - val json = gson.toJson(map) - val requestPersistTask = gson.fromJson(json, classOf[RequestPersistTask]) - requestPersistTask - case _ => throw AppJointErrorException(s"query from jobhistory not a correct RequestPersistTask type taskId is $jobId") - } - case _ => throw AppJointErrorException(s"query from jobhistory not a correct List type taskId is $jobId") - } - } - case _ => logger.error("get query response incorrectly") - throw AppJointErrorException("get query response incorrectly") - } - }{ - case errorException:ErrorException => throw errorException - case e:Exception => val e1 = AppJointErrorException(s"query taskId $jobId error") - e1.initCause(e) - throw e - } - task - } - - override def appendLog(log: String): Unit = { - entranceContext.getOrCreateLogManager().onLogUpdate(job, log) - } - - override def updateProgress(progress: Float): Unit = { - entranceContext.getOrCreatePersistenceManager().onProgressUpdate(job, progress, Array.empty) - } - - override def setStorePath(storePath: String): Unit = this.storePath = storePath - - override def getStorePath: String = this.storePath - - private def getMaxCache:Long = ByteTimeUtils.byteStringAsBytes("0K") - - override def createTableResultSetWriter(): ResultSetWriter[TableMetaData, TableRecord] = { - createTableResultSetWriter(defaultResultSetAlias) - } - - - - override def createTableResultSetWriter(resultSetAlias: String): ResultSetWriter[TableMetaData, TableRecord] = { - val resultSetType = ResultSetFactory.resultSetType.getOrElse(ResultSetFactory.TABLE_TYPE, "TABLE") - val resultSet = resultSetFactory.getResultSetByType(ResultSetFactory.TABLE_TYPE) - val filePath = this.storePath - val fileName = if(StringUtils.isEmpty(resultSetAlias)) "_" + aliasNum.getAndIncrement() else resultSetAlias + "_" + aliasNum.getAndIncrement() - val resultSetPath = resultSet.getResultSetPath(new FsPath(filePath), fileName) - val resultSetWriter = ResultSetWriter.getResultSetWriter(resultSet, getMaxCache, resultSetPath) - resultSetWriter.asInstanceOf[StorageResultSetWriter[TableMetaData, TableRecord]].setProxyUser(this.getUser) - resultSetWriter match { - case r:ResultSetWriter[TableMetaData, TableRecord] => r - case _ => null - } - } - - override def createTextResultSetWriter(): ResultSetWriter[LineMetaData, LineRecord] = { - createTextResultSetWriter(defaultResultSetAlias) - } - - override def createTextResultSetWriter(resultSetAlias: String): ResultSetWriter[LineMetaData, LineRecord] = { - val resultSetType = ResultSetFactory.resultSetType.getOrElse(ResultSetFactory.TEXT_TYPE, "TEXT") - val resultSet = resultSetFactory.getResultSetByType(ResultSetFactory.TEXT_TYPE) - val filePath = this.storePath - val fileName = if(StringUtils.isEmpty(resultSetAlias)) "_" + aliasNum.getAndIncrement() else resultSetAlias + "_" + aliasNum.getAndIncrement() - val resultSetPath = resultSet.getResultSetPath(new FsPath(filePath), fileName) - val resultSetWriter = ResultSetWriter.getResultSetWriter(resultSet, getMaxCache, resultSetPath) - resultSetWriter.asInstanceOf[StorageResultSetWriter[LineMetaData, LineRecord]].setProxyUser(this.getUser) - resultSetWriter match { - case r:ResultSetWriter[LineMetaData, LineRecord] => r - case _ => null - } - } - - override def createHTMLResultSetWriter(): ResultSetWriter[LineMetaData, LineRecord] = { - createHTMLResultSetWriter(defaultResultSetAlias) - } - - override def createHTMLResultSetWriter(resultSetAlias: String): ResultSetWriter[LineMetaData, LineRecord] = { - val resultSetType = ResultSetFactory.resultSetType.getOrElse(ResultSetFactory.HTML_TYPE, "HTML") - val resultSet = resultSetFactory.getResultSetByType(ResultSetFactory.HTML_TYPE) - val filePath = this.storePath - val fileName = if(StringUtils.isEmpty(resultSetAlias)) "_" + aliasNum.getAndIncrement() else resultSetAlias + "_" + aliasNum.getAndIncrement() - val resultSetPath = resultSet.getResultSetPath(new FsPath(filePath), fileName) - val resultSetWriter = ResultSetWriter.getResultSetWriter(resultSet, getMaxCache, resultSetPath) - resultSetWriter.asInstanceOf[StorageResultSetWriter[LineMetaData, LineRecord]].setProxyUser(this.getUser) - resultSetWriter match { - case r:ResultSetWriter[LineMetaData, LineRecord] => r - case _ => null - } - } - - override def createPictureResultSetWriter(): ResultSetWriter[LineMetaData, LineRecord] = { - createPictureResultSetWriter(defaultResultSetAlias) - } - - override def createPictureResultSetWriter(resultSetAlias: String): ResultSetWriter[LineMetaData, LineRecord] = { - val resultSetType = ResultSetFactory.resultSetType.getOrElse(ResultSetFactory.PICTURE_TYPE, "PICTURE") - val resultSet = resultSetFactory.getResultSetByType(ResultSetFactory.PICTURE_TYPE) - val filePath = this.storePath - val fileName = if(StringUtils.isEmpty(resultSetAlias)) "_" + aliasNum.getAndIncrement() else resultSetAlias + "_" + aliasNum.getAndIncrement() - val resultSetPath = resultSet.getResultSetPath(new FsPath(filePath), fileName) - val resultSetWriter = ResultSetWriter.getResultSetWriter(resultSet, getMaxCache, resultSetPath) - resultSetWriter.asInstanceOf[StorageResultSetWriter[LineMetaData, LineRecord]].setProxyUser(this.getUser) - resultSetWriter match { - case r:ResultSetWriter[LineMetaData, LineRecord] => r - case _ => null - } - } - - override def createResultSetWriter(resultSet: ResultSet[_ <: MetaData, _ <: Record], resultSetAlias: String): ResultSetWriter[_ <: MetaData, _ <: Record] = ??? - - //todo - override def getResultSetReader(fsPath: FsPath): ResultSetReader[_ <: MetaData, _ <: Record] = { - ResultSetReader.getResultSetReader(fsPath.getPath) - } - - /** - * - * @return - */ - override def getGatewayUrl: String = { - val instances = Utils.tryCatch{ - Sender.getInstances(AppJointEntranceConfiguration.GATEWAY_SPRING_APPLICATION.getValue) - }{ - case e:Exception => val e1 = RPCFailedException("获取gateway的url失败") - e1.initCause(e) - throw e1 - } - if (instances.length == 0) throw RPCFailedException("获取gateway的url失败") - instances(0).getInstance - } - - override def getResultSetPathsByJobId(jobId: Long): Array[FsPath] = { - val paths = new ArrayBuffer[FsPath]() - val task = getJobById(jobId) - val resultSetLocation = task.getResultLocation - val user = task.getUmUser - val fileSystem = FSFactory.getFsByProxyUser(new FsPath(resultSetLocation), user) - fileSystem.init(new util.HashMap[String, String]()) - Utils.tryFinally{ - fileSystem.list(new FsPath(resultSetLocation)) foreach (paths += _) - }{ - Utils.tryQuietly(fileSystem.close()) - } - paths.toArray - } -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/job/AppJointEntranceJob.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/job/AppJointEntranceJob.scala deleted file mode 100644 index f993cda8a..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/job/AppJointEntranceJob.scala +++ /dev/null @@ -1,100 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.job - -import java.util - -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, CommonAppJointNode} -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.execute.AppJointEntranceEngine -import com.webank.wedatasphere.linkis.common.exception.ErrorException -import com.webank.wedatasphere.linkis.common.log.LogUtils -import com.webank.wedatasphere.linkis.common.utils.Utils -import com.webank.wedatasphere.linkis.entrance.execute.StorePathExecuteRequest -import com.webank.wedatasphere.linkis.entrance.job.EntranceExecutionJob -import com.webank.wedatasphere.linkis.protocol.query.RequestPersistTask -import com.webank.wedatasphere.linkis.scheduler.executer._ -import com.webank.wedatasphere.linkis.scheduler.queue.SchedulerEventState._ -import org.apache.commons.lang.StringUtils -/** - * created by cooperyang on 2019/9/26 - * Description: - */ -class AppJointEntranceJob extends EntranceExecutionJob{ - var node:AppJointNode = _ - var runTimeMap:util.Map[String, Object] = _ - - def setNode(node:AppJointNode):Unit = this.node = node - def getNode:AppJointNode = this.node - - def setRunTimeMap(runTimeMap:util.Map[String, Object]):Unit = this.runTimeMap = runTimeMap - def getRunTimeMap:util.Map[String, Object] = this.runTimeMap - - - override def jobToExecuteRequest(): ExecuteRequest = new ExecuteRequest with StorePathExecuteRequest with AppJointExecuteRequest { - - - override val code: String = "" - - val jobContent: util.Map[String, Object] = AppJointEntranceJob.this.getNode match { - case commonAppJointNode:CommonAppJointNode => commonAppJointNode.getJobContent - case _ => null - } - override val storePath: String = AppJointEntranceJob.this.getTask match{ - case requestPersistTask:RequestPersistTask => requestPersistTask.getResultLocation - case _ => null - } - override val node: AppJointNode = AppJointEntranceJob.this.node - override val runTimeParams: util.Map[String, Object] = AppJointEntranceJob.this.runTimeMap - } - - - override def run(): Unit = { - if(!isScheduled) return - info(s"$getId starts to run") - getLogListener.foreach(_.onLogUpdate(this, LogUtils.generateInfo(s"$getId starts to execute."))) - startTime = System.currentTimeMillis - Utils.tryAndErrorMsg(transition(Running))(s"transition $getId from Scheduler to Running failed.") - getExecutor match { - case appjointEntranceEngine:AppJointEntranceEngine => appjointEntranceEngine.setJob(this) - } - val rs = Utils.tryCatch(getExecutor.execute(jobToExecuteRequest())){ - case e:ErrorException => logger.error(s"execute job $getId failed", e) - ErrorExecuteResponse(s"execute job $getId failed", e) - case t:Throwable => logger.error(s"execute job $getId failed", t) - ErrorExecuteResponse(s"execute job $getId failed", t) - } - rs match { - case r: CompletedExecuteResponse => - setResultSize(0) - transitionCompleted(r) - case r: IncompleteExecuteResponse => - setResultSize(0) - transitionCompleted(ErrorExecuteResponse(if(StringUtils.isNotEmpty(r.message)) r.message else "incomplete code.", null)) - case r: AsynReturnExecuteResponse => - setResultSize(0) - r.notify(r1 => { - val realRS = r1 match { - case r: IncompleteExecuteResponse => - ErrorExecuteResponse(if(StringUtils.isNotEmpty(r.message)) r.message else "incomplete code.", null) - case r: CompletedExecuteResponse => r - } - transitionCompleted(realRS) - }) - } - } -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/job/AppJointExecuteRequest.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/job/AppJointExecuteRequest.scala deleted file mode 100644 index 5c7d94812..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/job/AppJointExecuteRequest.scala +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.job - -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode - -/** - * created by cooperyang on 2019/9/26 - * Description: - */ -trait AppJointExecuteRequest { - val node:AppJointNode - val runTimeParams:java.util.Map[String, Object] - - override def toString: String = { - node.getProjectName + " " + node.getFlowName + " " + node.getName - } -} diff --git a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/parser/AppJointEntranceParser.scala b/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/parser/AppJointEntranceParser.scala deleted file mode 100644 index d5cd3e13c..000000000 --- a/plugins/linkis/linkis-appjoint-entrance/src/main/scala/com/webank/wedatasphere/dss/linkis/appjoint/entrance/parser/AppJointEntranceParser.scala +++ /dev/null @@ -1,95 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.linkis.appjoint.entrance.parser - -import java.util - -import com.webank.wedatasphere.dss.appjoint.execution.core.CommonAppJointNode -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.conf.AppJointConst -import com.webank.wedatasphere.dss.linkis.appjoint.entrance.job.AppJointEntranceJob -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import com.webank.wedatasphere.linkis.entrance.parser.CommonEntranceParser -import com.webank.wedatasphere.linkis.protocol.query.RequestPersistTask -import com.webank.wedatasphere.linkis.protocol.task.Task -import com.webank.wedatasphere.linkis.protocol.utils.TaskUtils -import com.webank.wedatasphere.linkis.scheduler.queue.Job - -/** - * created by cooperyang on 2019/9/26 - * Description: - */ - -class AppJointEntranceParser extends CommonEntranceParser with Logging{ - - private val PROJECT_NAME_STR = "projectName" - - private val FLOW_NAME_STR = "flowName" - - private val NODE_NAME_STR = "nodeName" - - private val TYPE_NAME_STR = "type" - - - override def parseToJob(task: Task): Job = { - val job = new AppJointEntranceJob - task match{ - case requestPersistTask:RequestPersistTask => - job.setTask(task) - job.setUser(requestPersistTask.getUmUser) - job.setCreator(requestPersistTask.getRequestApplicationName) - job.setParams(requestPersistTask.getParams.asInstanceOf[util.Map[String, Any]]) - val projectName = Utils.tryCatch{ - requestPersistTask.getSource.get(PROJECT_NAME_STR) - }{ - case e:Exception => "" - } - val flowName = Utils.tryCatch{ - requestPersistTask.getSource.get(FLOW_NAME_STR) - }{ - case e:Exception => "" - } - val nodeName = Utils.tryCatch{ - requestPersistTask.getSource.get(NODE_NAME_STR) - }{ - case e:Exception => "" - } - val _type = Utils.tryCatch{ - requestPersistTask.getRunType - }{ - case e:Exception => "" - } - val executionCode = requestPersistTask.getExecutionCode - val jobContent = AppJointConst.gson.fromJson(executionCode, classOf[util.Map[String,Object ]]) - val node = new CommonAppJointNode() - node.setProjectName(projectName) - node.setFlowName(flowName) - node.setNodeType(_type) - node.setJobContent(jobContent) - job.setNode(node) - val um_user = requestPersistTask.getUmUser - val runTimeMap = TaskUtils.getRuntimeMap(requestPersistTask.getParams.asInstanceOf[util.Map[String, Any]]) - runTimeMap.put("user", um_user) - runTimeMap.put("storePath", requestPersistTask.getResultLocation) - job.setRunTimeMap(runTimeMap.asInstanceOf[java.util.Map[String, Object]]) - job.setEntranceListenerBus(getEntranceContext.getOrCreateEventListenerBus) - job.setListenerEventBus(null) - job.setProgress(0f) - } - job - } -} diff --git a/pom.xml b/pom.xml index 53346c0ff..19e2f4e7d 100644 --- a/pom.xml +++ b/pom.xml @@ -1,9 +1,8 @@ - - - dss-qualitis-appjoint - - zip - - true - qualitis - - - - - - lib - true - true - false - true - true - - - - - - ${basedir}/src/main/resources - - appjoint.properties - - 0777 - / - unix - - - - ${basedir}/src/main/resources - - log4j.properties - log4j2.xml - - 0777 - conf - unix - - - - - - diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/Md5Utils.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/Md5Utils.java deleted file mode 100644 index 4d5abe6eb..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/Md5Utils.java +++ /dev/null @@ -1,27 +0,0 @@ -package com.webank.wedatasphere; - -import org.apache.commons.lang.StringUtils; - -import java.math.BigInteger; -import java.security.MessageDigest; - -/** - * @author howeye - */ -public class Md5Utils { - - public static String getMd5L32(String str) throws Exception { - try { - // Generate a instance of md5 - MessageDigest md = MessageDigest.getInstance("MD5"); - // calculate - md.update(str.getBytes()); - String result = new BigInteger(1, md.digest()).toString(16); - - return StringUtils.leftPad(result, 32, '0'); - } catch (Exception e) { - throw new Exception("Error generating md5 of string: " + str, e); - } - } - -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisAppJoint.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisAppJoint.java deleted file mode 100644 index 42f1b30b5..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisAppJoint.java +++ /dev/null @@ -1,56 +0,0 @@ -package com.webank.wedatasphere.appjoint; - -import com.webank.wedatasphere.project.QualitisProjectServiceImpl; -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.Map; - -/** - * @author howeye - */ -public class QualitisAppJoint implements AppJoint { - - QualitisNodeExecution nodeExecution = null; - QualitisProjectServiceImpl projectService = null; - - private static final Logger LOGGER = LoggerFactory.getLogger(QualitisAppJoint.class); - - @Override - public String getAppJointName() { - return "Qualitis"; - } - - @Override - public void init(String s, Map map) throws AppJointErrorException { - nodeExecution = new QualitisNodeExecution(); - nodeExecution.setBaseUrl(s); - nodeExecution.init(map); - - projectService = new QualitisProjectServiceImpl(); - projectService.setBaseUrl(s); - projectService.init(map); - } - - @Override - public NodeExecution getNodeExecution() { - if (nodeExecution != null) { - return nodeExecution; - } - LOGGER.error("NodeExecution is not inited, return null"); - return null; - } - - @Override - public ProjectService getProjectService() { - if (projectService != null) { - return projectService; - } - LOGGER.error("ProjectService is not inited, return null"); - return null; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisNodeExecution.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisNodeExecution.java deleted file mode 100644 index f2b0bbc38..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisNodeExecution.java +++ /dev/null @@ -1,326 +0,0 @@ -package com.webank.wedatasphere.appjoint; - -import com.google.gson.Gson; -import com.webank.wedatasphere.Md5Utils; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.common.CompletedNodeExecutionResponse; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionState; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.execution.core.LongTermNodeExecution; -import com.webank.wedatasphere.dss.appjoint.execution.core.NodeContext; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.node.Node; -import org.apache.commons.lang.RandomStringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.http.HttpEntity; -import org.springframework.http.HttpHeaders; -import org.springframework.http.HttpMethod; -import org.springframework.http.MediaType; -import org.springframework.web.client.RestTemplate; - -import javax.ws.rs.core.UriBuilder; -import java.net.URI; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -/** - * @author howeye - */ -public class QualitisNodeExecution extends LongTermNodeExecution { - - private static final String SUBMIT_TASK_PATH = "/qualitis/outer/api/v1/execution"; - private static final String GET_TASK_STATUS_PATH = "/qualitis/outer/api/v1/application/{applicationId}/status/"; - private static final String GET_TASK_RESULT_PATH = "/qualitis/outer/api/v1/application/{applicationId}/result/"; - - private static final Logger LOGGER = LoggerFactory.getLogger(QualitisAppJoint.class); - - private String host = null; - private Integer port = null; - private String appId = null; - private String appToken = null; - - private String baseUrl; - - public boolean canExecute(Node node, NodeContext nodeContext) { - return true; - } - - @Override - public void init(Map map) throws AppJointErrorException { - appId = (String) map.get("qualitis_appId"); - appToken = (String) map.get("qualitis_appToken"); - host = getHost(); - port = getPort(); - } - - @Override - public boolean canExecute(AppJointNode appJointNode, NodeContext context, Session session) { - return appJointNode.getNodeType().toLowerCase().contains("qualitis"); - } - - @Override - public NodeExecutionAction submit(AppJointNode appJointNode, NodeContext nodeContext, Session session) { - try { - Map map = nodeContext.getRuntimeMap(); - String filter = (String) map.get("filter"); - String executionUser = (String) map.get("executeUser"); - String createUser = (String) map.get("user"); - Map map1 = ((AppJointNode)appJointNode).getJobContent(); - String id = map1.get("ruleGroupId").toString(); - float f = Float.valueOf(id); - Long groupId = (long)f; - QualitisSubmitRequest submitRequest = new QualitisSubmitRequest(); - submitRequest.setCreateUser(createUser); - submitRequest.setExecutionUser(executionUser); - submitRequest.setPartition(filter); - submitRequest.setGroupId(groupId); - - RestTemplate restTemplate = new RestTemplate(); - HttpEntity entity = generateEntity(submitRequest); - - URI url = buildUrI(host, port, SUBMIT_TASK_PATH, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - String startLog = String.format("Start to submit job to qualitis. url: %s, method: %s, body: %s", url, javax.ws.rs.HttpMethod.POST, entity); - LOGGER.info(startLog); - nodeContext.appendLog(startLog); - Map response = restTemplate.postForObject(url, entity, Map.class); - String finishLog = String.format("Succeed to submit job to qualitis. response: %s", response); - LOGGER.info(finishLog); - nodeContext.appendLog(finishLog); - - if (response == null) { - String errorMsg = "Error! Can not submit job, response is null"; - LOGGER.error(errorMsg); - nodeContext.appendLog(errorMsg); - return null; - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - String errorMsg = String.format("Error! Can not submit job, exception: %s", message); - LOGGER.error(errorMsg); - nodeContext.appendLog(errorMsg); - return null; - } - - String applicationId = (String) ((Map) response.get("data")).get("application_id"); - return new QualitisNodeExecutionAction(applicationId); - } catch (Exception e) { - String errorMsg = "Error! Can not submit job"; - LOGGER.error(errorMsg, e); - nodeContext.appendLog(errorMsg); - return null; - } - } - - @Override - public NodeExecutionState state(NodeExecutionAction nodeExecutionAction) { - if (nodeExecutionAction == null) { - return NodeExecutionState.Failed; - } - - String applicationId = ((QualitisNodeExecutionAction)nodeExecutionAction).getApplicationId(); - try { - // 发送请求查看任务的状态 - RestTemplate restTemplate = new RestTemplate(); - HttpHeaders headers = new HttpHeaders(); - HttpEntity entity = new HttpEntity(headers); - - String path = GET_TASK_STATUS_PATH.replace("{applicationId}", applicationId); - URI url = buildUrI(host, port, path, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - LOGGER.info("Start to get job status. url: {}, method: {}, body: {}", url, HttpMethod.GET, entity); - Map response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody(); - LOGGER.info("Succeed to get job status. response: {}", response); - - if (response == null) { - LOGGER.error("Error! Can not get job status, job_id: {}, response is null", applicationId); - return NodeExecutionState.Failed; - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - LOGGER.error("Error! Can not get job status, exception: {}", message); - return NodeExecutionState.Failed; - } - - List> tasks = (List>) ((Map) response.get("data")).get("task"); - Map statusCountMap = new HashMap(8); - initCountMap(statusCountMap); - Integer taskSize = tasks.size(); - for (Map task : tasks) { - Integer taskStatus = (Integer) task.get("task_status"); - addStatus(taskStatus, statusCountMap); - } - - Integer runningCount = statusCountMap.get(NodeExecutionState.Running); - Integer successCount = statusCountMap.get(NodeExecutionState.Success); - Integer failedCount = statusCountMap.get(NodeExecutionState.Failed); - - if (runningCount != 0) { - return NodeExecutionState.Running; - } else if (successCount.equals(taskSize)) { - return NodeExecutionState.Success; - } else if (failedCount != 0) { - return NodeExecutionState.Failed; - } else { - return NodeExecutionState.Accepted; - } - } catch (Exception e) { - LOGGER.error("Error! Can not get job status, job_id: {}", applicationId, e); - return NodeExecutionState.Failed; - } - } - - @Override - public CompletedNodeExecutionResponse result(NodeExecutionAction nodeExecutionAction, NodeContext nodeContext) { - if (nodeExecutionAction == null) { - return null; - } - - String applicationId = ((QualitisNodeExecutionAction)nodeExecutionAction).getApplicationId(); - try { - // Send request and get response - RestTemplate restTemplate = new RestTemplate(); - HttpHeaders headers = new HttpHeaders(); - HttpEntity entity = new HttpEntity(headers); - - String path = GET_TASK_RESULT_PATH.replace("{applicationId}", applicationId); - URI url = buildUrI(host, port, path, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - - String startLog = String.format("Start to get job result. url: %s, method: %s, body: %s", url, HttpMethod.GET, entity); - LOGGER.info(startLog); - nodeContext.appendLog(startLog); - Map response = restTemplate.exchange(url, HttpMethod.GET, entity, Map.class).getBody(); - String finishLog = String.format("Succeed to get job result. response: %s", response); - LOGGER.info(finishLog); - nodeContext.appendLog(finishLog); - - if (response == null) { - String errorMsg = String.format("Error! Can not get job result, job_id: %s, response is null", applicationId); - LOGGER.error(errorMsg); - nodeContext.appendLog(errorMsg); - return null; - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - String errorMsg = String.format("Error! Can not get job result, exception: {}", message); - LOGGER.error(errorMsg); - nodeContext.appendLog(errorMsg); - return null; - } - - Integer passNum = (Integer) ((Map) response.get("data")).get("pass_num"); - Integer failedNum = (Integer) ((Map) response.get("data")).get("failed_num"); - Integer notPassNum = (Integer) ((Map) response.get("data")).get("not_pass_num"); - String resultMessage = (String) ((Map) response.get("data")).get("result_message"); - - String taskMsg = String.format("Task result: Pass/Failed/Not Pass ------- %s/%s/%s", passNum, failedNum, notPassNum); - LOGGER.info(taskMsg); - LOGGER.info(resultMessage); - nodeContext.appendLog(taskMsg); - nodeContext.appendLog(resultMessage); - - CompletedNodeExecutionResponse result = new CompletedNodeExecutionResponse(); - if (failedNum != 0 || notPassNum != 0) { - result.setIsSucceed(false); - } else { - result.setIsSucceed(true); - } - return result; - } catch (Exception e) { - String errorMsg = String.format("Error! Can not get job result, job_id: %s", applicationId); - LOGGER.error(errorMsg, e); - nodeContext.appendLog(errorMsg); - return null; - } - } - - private void addStatus(Integer status, Map statusCountMap) { - if (status.equals(QualitisTaskStatusEnum.SUBMITTED.getCode())) { - statusCountMap.put(NodeExecutionState.Accepted, statusCountMap.get(NodeExecutionState.Accepted) + 1); - } else if (status.equals(QualitisTaskStatusEnum.INITED.getCode())) { - statusCountMap.put(NodeExecutionState.Accepted, statusCountMap.get(NodeExecutionState.Accepted) + 1); - } else if (status.equals(QualitisTaskStatusEnum.RUNNING.getCode())) { - statusCountMap.put(NodeExecutionState.Running, statusCountMap.get(NodeExecutionState.Running) + 1); - } else if (status.equals(QualitisTaskStatusEnum.SUCCEED.getCode())) { - statusCountMap.put(NodeExecutionState.Success, statusCountMap.get(NodeExecutionState.Success) + 1); - } else if (status.equals(QualitisTaskStatusEnum.PASS_CHECKOUT.getCode())) { - statusCountMap.put(NodeExecutionState.Success, statusCountMap.get(NodeExecutionState.Success) + 1); - } else if (status.equals(QualitisTaskStatusEnum.FAIL_CHECKOUT.getCode())) { - statusCountMap.put(NodeExecutionState.Failed, statusCountMap.get(NodeExecutionState.Failed) + 1); - } else if (status.equals(QualitisTaskStatusEnum.FAILED.getCode())) { - statusCountMap.put(NodeExecutionState.Failed, statusCountMap.get(NodeExecutionState.Failed) + 1); - } else if (status.equals(QualitisTaskStatusEnum.TASK_NOT_EXIST.getCode())) { - statusCountMap.put(NodeExecutionState.Failed, statusCountMap.get(NodeExecutionState.Failed) + 1); - } else if (status.equals(QualitisTaskStatusEnum.CANCELLED.getCode())) { - statusCountMap.put(NodeExecutionState.Killed, statusCountMap.get(NodeExecutionState.Killed) + 1); - } else if (status.equals(QualitisTaskStatusEnum.TIMEOUT.getCode())) { - statusCountMap.put(NodeExecutionState.Failed, statusCountMap.get(NodeExecutionState.Failed) + 1); - } else if (status.equals(QualitisTaskStatusEnum.SCHEDULED.getCode())) { - statusCountMap.put(NodeExecutionState.Accepted, statusCountMap.get(NodeExecutionState.Accepted) + 1); - } - } - - private void initCountMap(Map statusCountMap) { - statusCountMap.put(NodeExecutionState.Accepted, 0); - statusCountMap.put(NodeExecutionState.Running, 0); - statusCountMap.put(NodeExecutionState.Success, 0); - statusCountMap.put(NodeExecutionState.Failed, 0); - - } - - private Boolean checkResponse(Map response) { - String responseStatus = (String) response.get("code"); - return "200".equals(responseStatus); - } - - private HttpEntity generateEntity(Object submitRequest) { - HttpHeaders headers = new HttpHeaders(); - headers.setContentType(MediaType.APPLICATION_JSON); - - Gson gson = new Gson(); - return new HttpEntity(gson.toJson(submitRequest), headers); - } - - private URI buildUrI(String host, int port, String path, String appId, String appToken, - String nonce, String timestamp) throws Exception { - String signature = getSignature(appId, appToken, nonce, timestamp); - String urlStr = "http://" + host + ":" + port; - URI uri = UriBuilder.fromUri(urlStr) - .path(path) - .queryParam("app_id", appId) - .queryParam("nonce", nonce) - .queryParam("timestamp", timestamp) - .queryParam("signature", signature) - .build(); - return uri; - } - - private String getSignature(String appId, String appToken, String nonce, String timestamp) - throws Exception { - return Md5Utils.getMd5L32(Md5Utils.getMd5L32(appId + nonce + timestamp) + appToken); - } - - private Integer getPort() { - String baseUrl = getBaseUrl(); - return UriBuilder.fromUri(baseUrl).build().getPort(); - } - - private String getHost() { - String baseUrl = getBaseUrl(); - return UriBuilder.fromUri(baseUrl).build().getHost(); - } - - @Override - public String getBaseUrl() { - return this.baseUrl; - } - - @Override - public void setBaseUrl(String s) { - this.baseUrl = s; - } -} \ No newline at end of file diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisNodeExecutionAction.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisNodeExecutionAction.java deleted file mode 100644 index a42b4733d..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisNodeExecutionAction.java +++ /dev/null @@ -1,19 +0,0 @@ -package com.webank.wedatasphere.appjoint; - -import com.webank.wedatasphere.dss.appjoint.execution.common.NodeExecutionAction; - -/** - * @author howeye - */ -public class QualitisNodeExecutionAction implements NodeExecutionAction { - - private String applicationId; - - public QualitisNodeExecutionAction(String applicationId) { - this.applicationId = applicationId; - } - - public String getApplicationId() { - return applicationId; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisSubmitRequest.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisSubmitRequest.java deleted file mode 100644 index 180da4437..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisSubmitRequest.java +++ /dev/null @@ -1,55 +0,0 @@ -package com.webank.wedatasphere.appjoint; - -import java.util.List; -import com.google.gson.annotations.SerializedName; - -/** - * @author howeye - */ -public class QualitisSubmitRequest { - - @SerializedName("group_id") - private Long groupId; - - private String partition; - @SerializedName("execution_user") - private String executionUser; - - @SerializedName("create_user") - private String createUser; - - public QualitisSubmitRequest() { - } - - public Long getGroupId() { - return groupId; - } - - public void setGroupId(Long groupId) { - this.groupId = groupId; - } - - public String getPartition() { - return partition; - } - - public void setPartition(String partition) { - this.partition = partition; - } - - public String getExecutionUser() { - return executionUser; - } - - public void setExecutionUser(String executionUser) { - this.executionUser = executionUser; - } - - public String getCreateUser() { - return createUser; - } - - public void setCreateUser(String createUser) { - this.createUser = createUser; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisTaskStatusEnum.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisTaskStatusEnum.java deleted file mode 100644 index 904d87ecb..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/appjoint/QualitisTaskStatusEnum.java +++ /dev/null @@ -1,43 +0,0 @@ -package com.webank.wedatasphere.appjoint; - -/** - * @author howeye - */ -public enum QualitisTaskStatusEnum { - /** - * Task status enum - */ - SUBMITTED(1, "已提交", "SUBMITTED"), - INITED(2, "初始化", "INITED"), - RUNNING(3, "运行中", "RUNNING"), - SUCCEED(4, "成功", "SUCCEED"), - PASS_CHECKOUT(5, "通过校验", "PASS_CHECKOUT"), - FAIL_CHECKOUT(6, "未通过校验", "FAIL_CHECKOUT"), - FAILED(7, "失败", "FAILED"), - TASK_NOT_EXIST(8, "Task不存在", "TASK_NOT_EXIST"), - CANCELLED(9, "取消", "CANCELLED"), - TIMEOUT(10, "超时", "TIMEOUT"), - SCHEDULED(11, "调度中", "SCHEDULED"),; - - private Integer code; - private String message; - private String state; - - QualitisTaskStatusEnum(Integer code, String message, String state) { - this.code = code; - this.message = message; - this.state = state; - } - - public Integer getCode() { - return code; - } - - public String getMessage() { - return message; - } - - public String getState() { - return state; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisAddProjectRequest.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisAddProjectRequest.java deleted file mode 100644 index 1019f2056..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisAddProjectRequest.java +++ /dev/null @@ -1,41 +0,0 @@ -package com.webank.wedatasphere.project; - -import com.google.gson.annotations.SerializedName; - -/** - * @author howeye - */ -public class QualitisAddProjectRequest { - - @SerializedName("project_name") - private String projectName; - private String description; - private String username; - - public QualitisAddProjectRequest() { - } - - public String getProjectName() { - return projectName; - } - - public void setProjectName(String projectName) { - this.projectName = projectName; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public String getUsername() { - return username; - } - - public void setUsername(String username) { - this.username = username; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisDeleteProjectRequest.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisDeleteProjectRequest.java deleted file mode 100644 index d4909265a..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisDeleteProjectRequest.java +++ /dev/null @@ -1,32 +0,0 @@ -package com.webank.wedatasphere.project; - -import com.google.gson.annotations.SerializedName; - -/** - * @author howeye - */ -public class QualitisDeleteProjectRequest { - - @SerializedName("project_id") - private Long projectId; - private String username; - - public QualitisDeleteProjectRequest() { - } - - public Long getProjectId() { - return projectId; - } - - public void setProjectId(Long projectId) { - this.projectId = projectId; - } - - public String getUsername() { - return username; - } - - public void setUsername(String username) { - this.username = username; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisProjectServiceImpl.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisProjectServiceImpl.java deleted file mode 100644 index 0c3cde2b9..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisProjectServiceImpl.java +++ /dev/null @@ -1,261 +0,0 @@ -package com.webank.wedatasphere.project; - - -import com.google.gson.Gson; -import com.webank.wedatasphere.Md5Utils; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.appjoint.service.ProjectService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.common.entity.project.DSSProject; -import com.webank.wedatasphere.dss.common.entity.project.Project; -import org.apache.commons.lang.RandomStringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.http.HttpEntity; -import org.springframework.http.HttpHeaders; -import org.springframework.http.HttpMethod; -import org.springframework.http.MediaType; -import org.springframework.web.client.RestTemplate; - -import javax.ws.rs.core.UriBuilder; -import java.net.URI; -import java.util.List; -import java.util.Map; - -/** - * @author howeye - */ -public class QualitisProjectServiceImpl extends AppJointUrlImpl implements ProjectService { - - private static final String CREATE_PROJECT_PATH = "/qualitis/outer/api/v1/project/workflow"; - private static final String DELETE_PROJECT_PATH = "/qualitis/outer/api/v1/project/workflow/delete"; - private static final String UPDATE_PROJECT_PATH = "/qualitis/outer/api/v1/project/workflow"; - private static final String AUTO_ADD_USER_PATH = "/qualitis/outer/api/v1/user/{username}"; - - private String appId = "linkis_id"; - private String appToken = "a33693de51"; - - private String host = "localhost"; - private Integer port = 8090; - - private static final String FAILURE_CODE = "400"; - - private static final Logger LOGGER = LoggerFactory.getLogger(QualitisProjectServiceImpl.class); - - public void init(Map map) throws AppJointErrorException { - appId = (String) map.get("qualitis_appId"); - appToken = (String) map.get("qualitis_appToken"); - host = getHost(); - port = getPort(); - } - - @Override - public List fetchProjects(Session session) throws AppJointErrorException { - return null; - } - - @Override - public Project createProject(Project project, Session session) throws AppJointErrorException { - try { - QualitisAddProjectRequest qualitisAddProjectRequest = new QualitisAddProjectRequest(); - - DSSProject dssProject = (DSSProject) project; - qualitisAddProjectRequest.setProjectName(project.getName()); - qualitisAddProjectRequest.setDescription(project.getDescription()); - qualitisAddProjectRequest.setUsername(dssProject.getUserName()); - - RestTemplate restTemplate = new RestTemplate(); - HttpEntity entity = generateEntity(qualitisAddProjectRequest); - - Map response = createProjectReal(restTemplate, entity); - - String responseStatus = (String) response.get("code"); - if (FAILURE_CODE.equals(responseStatus)) { - // Send request to auto create qualitis user - autoAddUser(restTemplate, dssProject.getUserName()); - - // restart to create project - response = createProjectReal(restTemplate, entity); - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - String errorMessage = String.format("Error! Can not add project, exception: %s", message); - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - Long projectId = Long.valueOf(String.valueOf(((Map) ((Map) response.get("data")).get("project_detail")).get("project_id"))); - project.setId(projectId); - - LOGGER.info("Succeed to add project to qualitis, project_id: {}", project.getId()); - return project; - } catch (Exception e) { - LOGGER.error("Failed to add project to qualitis, caused by: {}", e.getMessage(), e); - throw new AppJointErrorException(500, e.getMessage()); - } - } - - private void autoAddUser(RestTemplate restTemplate, String username) throws Exception { - String path = AUTO_ADD_USER_PATH.replace("{username}", username); - URI url = buildUrI(getHost(), getPort(), path, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - - HttpHeaders headers = new HttpHeaders(); - HttpEntity entity = new HttpEntity(headers); - - LOGGER.info("Start to add user in qualitis. url: {}, method: {}", url, HttpMethod.POST); - Map response = restTemplate.exchange(url, org.springframework.http.HttpMethod.POST, entity, Map.class).getBody(); - LOGGER.info("Succeed to add user in qualitis. response: {}", response); - - if (response == null) { - String errorMessage = "Error! Can not add user, response is null"; - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - String errorMessage = String.format("Error! Can not add user, exception: %s", message); - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - } - - private Map createProjectReal(RestTemplate restTemplate, HttpEntity entity) throws Exception { - URI url = buildUrI(getHost(), getPort(), CREATE_PROJECT_PATH, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - LOGGER.info("Start to add project to qualitis. url: {}, method: {}, body: {}", url, HttpMethod.PUT, entity); - Map response = restTemplate.exchange(url, org.springframework.http.HttpMethod.PUT, entity, Map.class).getBody(); - LOGGER.info("Succeed to add project to qualitis. response: {}", response); - - if (response == null) { - String errorMessage = "Error! Can not add project, response is null"; - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - return response; - } - - @Override - public void deleteProject(Project project, Session session) throws AppJointErrorException { - try { - QualitisDeleteProjectRequest qualitisDeleteProjectRequest = new QualitisDeleteProjectRequest(); - - DSSProject dssProject = (DSSProject) project; - qualitisDeleteProjectRequest.setProjectId(project.getId()); - qualitisDeleteProjectRequest.setUsername(dssProject.getUserName()); - - RestTemplate restTemplate = new RestTemplate(); - HttpEntity entity = generateEntity(qualitisDeleteProjectRequest); - - URI url = buildUrI(getHost(), getPort(), DELETE_PROJECT_PATH, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - LOGGER.info("Start to delete project in qualitis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.POST, entity); - Map response = restTemplate.postForObject(url, entity, Map.class); - LOGGER.info("Succeed to delete project in qualitis. response: {}", response); - - if (response == null) { - String errorMessage = "Error! Can not delete project in qualitis, response is null"; - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - String errorMessage = String.format("Error! Can not delete project in qualitis, exception: %s", message); - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - LOGGER.info("Succeed to delete project in qualitis, project_id: {}", project.getId()); - } catch (Exception e) { - LOGGER.error("Failed to delete project in qualitis, caused by: {}", e.getMessage(), e); - throw new AppJointErrorException(500, e.getMessage()); - } - } - - public Project getProject(Project project) throws AppJointErrorException { - return null; - } - - @Override - public void updateProject(Project project, Session session) throws AppJointErrorException { - try { - QualitisUpdateProjectRequest qualitisUpdateProjectRequest = new QualitisUpdateProjectRequest(); - - DSSProject dssProject = (DSSProject) project; - qualitisUpdateProjectRequest.setProjectId(project.getId()); - qualitisUpdateProjectRequest.setProjectName(project.getName()); - qualitisUpdateProjectRequest.setDescription(project.getDescription()); - qualitisUpdateProjectRequest.setUsername(dssProject.getUserName()); - - RestTemplate restTemplate = new RestTemplate(); - HttpEntity entity = generateEntity(qualitisUpdateProjectRequest); - - URI url = buildUrI(getHost(), getPort(), UPDATE_PROJECT_PATH, appId, appToken, RandomStringUtils.randomNumeric(5), String.valueOf(System.currentTimeMillis())); - LOGGER.info("Start to update project in qualitis. url: {}, method: {}, body: {}", url, javax.ws.rs.HttpMethod.POST, entity); - Map response = restTemplate.postForObject(url, entity, Map.class); - LOGGER.info("Succeed to update project in qualitis. response: {}", response); - - if (response == null) { - String errorMessage = "Error! Can not update project in qualitis, response is null"; - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - if (!checkResponse(response)) { - String message = (String) response.get("message"); - String errorMessage = String.format("Error! Can not update project in qualitis, exception: %s", message); - LOGGER.error(errorMessage); - throw new AppJointErrorException(500, errorMessage); - } - - LOGGER.info("Succeed to update project in qualitis, project_id: {}", project.getId()); - } catch (Exception e) { - LOGGER.error("Failed to update project in qualitis, caused by: {}", e.getMessage(), e); - throw new AppJointErrorException(500, e.getMessage()); - } - } - - private Boolean checkResponse(Map response) { - String responseStatus = (String) response.get("code"); - return "200".equals(responseStatus); - } - - private HttpEntity generateEntity(Object submitRequest) { - HttpHeaders headers = new HttpHeaders(); - headers.setContentType(MediaType.APPLICATION_JSON); - - Gson gson = new Gson(); - return new HttpEntity(gson.toJson(submitRequest), headers); - } - - private URI buildUrI(String host, int port, String path, String appId, String appToken, - String nonce, String timestamp) throws Exception { - String signature = getSignature(appId, appToken, nonce, timestamp); - String urlStr = "http://" + host + ":" + port; - URI uri = UriBuilder.fromUri(urlStr) - .path(path) - .queryParam("app_id", appId) - .queryParam("nonce", nonce) - .queryParam("timestamp", timestamp) - .queryParam("signature", signature) - .build(); - return uri; - } - - private String getSignature(String appId, String appToken, String nonce, String timestamp) - throws Exception { - return Md5Utils.getMd5L32(Md5Utils.getMd5L32(appId + nonce + timestamp) + appToken); - } - - private Integer getPort() { - String baseUrl = getBaseUrl(); - return UriBuilder.fromUri(baseUrl).build().getPort(); - } - - private String getHost() { - String baseUrl = getBaseUrl(); - return UriBuilder.fromUri(baseUrl).build().getHost(); - } -} \ No newline at end of file diff --git a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisUpdateProjectRequest.java b/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisUpdateProjectRequest.java deleted file mode 100644 index cb95958bb..000000000 --- a/qualitis-appjoint/appjoint/src/main/java/com.webank/wedatasphere/project/QualitisUpdateProjectRequest.java +++ /dev/null @@ -1,51 +0,0 @@ -package com.webank.wedatasphere.project; - -import com.google.gson.annotations.SerializedName; - -/** - * @author howeye - */ -public class QualitisUpdateProjectRequest { - - @SerializedName("project_id") - private Long projectId; - @SerializedName("project_name") - private String projectName; - private String description; - private String username; - - public QualitisUpdateProjectRequest() { - } - - public Long getProjectId() { - return projectId; - } - - public void setProjectId(Long projectId) { - this.projectId = projectId; - } - - public String getProjectName() { - return projectName; - } - - public void setProjectName(String projectName) { - this.projectName = projectName; - } - - public String getDescription() { - return description; - } - - public void setDescription(String description) { - this.description = description; - } - - public String getUsername() { - return username; - } - - public void setUsername(String username) { - this.username = username; - } -} diff --git a/qualitis-appjoint/appjoint/src/main/resources/appjoint.properties b/qualitis-appjoint/appjoint/src/main/resources/appjoint.properties deleted file mode 100644 index 65fad6835..000000000 --- a/qualitis-appjoint/appjoint/src/main/resources/appjoint.properties +++ /dev/null @@ -1,19 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# -baseUrl=http://127.0.0.1:8090 -qualitis_appId=linkis_id -qualitis_appToken=a33693de51 diff --git a/qualitis-appjoint/appjoint/src/main/resources/log4j.properties b/qualitis-appjoint/appjoint/src/main/resources/log4j.properties deleted file mode 100644 index 0807e6087..000000000 --- a/qualitis-appjoint/appjoint/src/main/resources/log4j.properties +++ /dev/null @@ -1,37 +0,0 @@ -# -# Copyright 2019 WeBank -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO diff --git a/qualitis-appjoint/appjoint/src/main/resources/log4j2.xml b/qualitis-appjoint/appjoint/src/main/resources/log4j2.xml deleted file mode 100644 index 3923cd9f3..000000000 --- a/qualitis-appjoint/appjoint/src/main/resources/log4j2.xml +++ /dev/null @@ -1,39 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - diff --git a/sbin/common.sh b/sbin/common.sh new file mode 100644 index 000000000..2fd210ad0 --- /dev/null +++ b/sbin/common.sh @@ -0,0 +1,182 @@ +#!/bin/sh +# +# Copyright 2019 WeBank +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +#Actively load user env +source ~/.bash_profile + +export local_host="`hostname --fqdn`" + +ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}') + +function isLocal(){ + if [ "$1" == "127.0.0.1" ];then + return 0 + elif [ "$1" == "" ]; then + return 0 + elif [ "$1" == "localhost" ]; then + return 0 + elif [ "$1" == $local_host ]; then + return 0 + elif [ "$1" == $ipaddr ]; then + return 0 + fi + return 1 +} + +function executeCMD(){ + isLocal $1 + flag=$? + if [ $flag == "0" ];then + echo "Is local execution:$2" + eval $2 + else + echo "Is remote execution:$2" + ssh -p $SSH_PORT $1 $2 + fi + +} +function copyFile(){ + isLocal $1 + flag=$? + src=$2 + dest=$3 + if [ $flag == "0" ];then + echo "Is local cp " + cp -r "$src" "$dest" + else + echo "Is remote cp " + scp -r -P $SSH_PORT "$src" $1:"$dest" + fi + +} + +function isSuccess(){ +if [ $? -ne 0 ]; then + echo "Failed to " + $1 + exit 1 +else + echo "Succeed to" + $1 +fi +} + + +function start() +{ + echo "Start to check whether the $SERVER_NAME is running" + if [[ -f "${SERVER_PID}" ]]; then + pid=$(cat ${SERVER_PID}) + if kill -0 ${pid} >/dev/null 2>&1; then + echo "$SERVER_NAME is already running." + exit 1 + fi + fi + export SERVER_START_BIN=$DSS_HOME/sbin/ext/$SERVER_NAME + if [[ ! -f "${SERVER_START_BIN}" ]]; then + echo "The $SERVER_NAME is wrong or the corresponding startup script does not exist: " + echo "$SERVER_START_BIN" + exit 1 + else + echo "Start to start server, startup script: $SERVER_START_BIN" + sh $SERVER_START_BIN + fi +} + +function wait_for_server_to_die() { + local pid + local count + pid=$1 + timeout=$2 + count=0 + timeoutTime=$(date "+%s") + let "timeoutTime+=$timeout" + currentTime=$(date "+%s") + forceKill=1 + + while [[ $currentTime -lt $timeoutTime ]]; do + $(kill ${pid} > /dev/null 2> /dev/null) + if kill -0 ${pid} > /dev/null 2>&1; then + sleep 3 + else + forceKill=0 + break + fi + currentTime=$(date "+%s") + done + + if [[ forceKill -ne 0 ]]; then + $(kill -9 ${pid} > /dev/null 2> /dev/null) + fi +} + + +function stop() +{ + if [[ ! -f "${SERVER_PID}" ]]; then + echo "server $SERVER_NAME is not running" + else + pid=$(cat ${SERVER_PID}) + if [[ -z "${pid}" ]]; then + echo "server $SERVER_NAME is not running" + else + wait_for_server_to_die $pid 40 + $(rm -f ${SERVER_PID}) + echo "server $SERVER_NAME is stopped." + fi + fi +} + +function restart() +{ + stop + sleep 2 + start +} + +status() +{ + if [[ ! -f "${SERVER_PID}" ]]; then + echo "server $SERVER_NAME is stopped" + else + pid=$(cat ${SERVER_PID}) + if [[ -z "${pid}" ]]; then + echo "server $SERVER_NAME is not running" + else + echo "server $SERVER_NAME is running." + fi + fi +} + +function setServerName(){ + if [[ $PROJECT_NAME == *"project"* ]]; then + SERVER_NAME=dss-framework-project-server + elif [[ $PROJECT_NAME == *"orchestrator"* ]]; then + SERVER_NAME=dss-framework-orchestrator-server + elif [[ $PROJECT_NAME == *"apiservice"* ]]; then + SERVER_NAME=dss-apiservice-server + elif [[ $PROJECT_NAME == *"datapipe"* ]]; then + SERVER_NAME=dss-datapipe-server + elif [[ $PROJECT_NAME == *"workflow"* ]]; then + SERVER_NAME=dss-workflow-server + elif [[ $PROJECT_NAME == *"execution"* ]]; then + SERVER_NAME=dss-flow-execution-server + else + echo "please input： sh dss-daemon.sh [start,restart,stop] [server name]; for example : sh dss-daemon.sh restart project " + echo "server name : project、orchestrator、apiservice、datapipe、workflow、execution" + exit 1 + fi +} + diff --git a/sbin/dss-daemon.sh b/sbin/dss-daemon.sh new file mode 100644 index 000000000..a0c5d38a9 --- /dev/null +++ b/sbin/dss-daemon.sh @@ -0,0 +1,62 @@ +#!/bin/bash +# +# description: Starts and stops Server +# +# @name: dss-demo +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for dss 1.0.0 + + +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +function print_usage(){ + echo "Usage: dss-daemon [start | stop | restart | status] [serverName]" + echo " serverName The service name of the operation" + echo "Most commands print help when invoked w/o parameters." +} + +if [ $# != 2 ]; then + print_usage + exit 2 +fi + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +# get pid directory +if [ "$DSS_PID_DIR" = "" ]; then + export DSS_PID_DIR="$DSS_HOME/pid" +fi +if [ ! -w "$DSS_PID_DIR" ] ; then + mkdir -p "$DSS_PID_DIR" +fi +source $DSS_HOME/sbin/common.sh +source $DSS_HOME/conf/config.sh +typeset -l PROJECT_NAME +PROJECT_NAME=$2 + +## get project full name +setServerName + +COMMAND=$1 +export SERVER_PID=$DSS_PID_DIR/$SERVER_NAME.pid +case $COMMAND in + start|stop|restart|status) + $COMMAND $SERVER_NAME + ;; + *) + print_usage + exit 2 + ;; +esac \ No newline at end of file diff --git a/sbin/dss-start-all.sh b/sbin/dss-start-all.sh new file mode 100644 index 000000000..03dc12933 --- /dev/null +++ b/sbin/dss-start-all.sh @@ -0,0 +1,139 @@ +#!/usr/bin/env bash +# +# Copyright 2019 WeBank +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Start all dss applications +info="We will start all dss applications, it will take some time, please wait" +echo ${info} + +#Actively load user env + + +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +local_host="`hostname --fqdn`" +source $DSS_HOME/sbin/common.sh +source $DSS_HOME/conf/config.sh + +function startApp(){ +echo "<-------------------------------->" +echo "Begin to start $SERVER_NAME" +SERVER_START_CMD="sh $DSS_INSTALL_HOME/sbin/dss-daemon.sh restart $SERVER_NAME" +if test -z "$SERVER_IP" +then + SERVER_IP=$local_host +fi +executeCMD $SERVER_IP "$SERVER_START_CMD" +echo "End to start $SERVER_NAME" +echo "<-------------------------------->" +sleep 1 +} + +function checkServer() { +echo "<-------------------------------->" +echo "Begin to check $SERVER_NAME" +SERVER_CHECK_CMD="sh $DSS_HOME/sbin/dss-daemon.sh status $SERVER_NAME" +if test -z "$SERVER_IP" +then + SERVER_IP=$local_host +fi + +executeCMD $SERVER_IP "$SERVER_CHECK_CMD" + +if [ $? -ne 0 ]; then + ALL_SERVER_NAME=$SERVER_NAME + LOG_PATH=$DSS_HOME/logs/$ALL_SERVER_NAME.log + echo "ERROR: your $ALL_SERVER_NAME microservice is not start successful !!! ERROR logs as follows :" + echo "Please check detail log, log path :$LOG_PATH" + echo '<---------------------------------------------------->' + executeCMD $ALL_SERVER_NAME "tail -n 50 $LOG_PATH" + echo '<---------------------------------------------------->' + echo "Please check detail log, log path :$LOG_PATH" + exit 1 +fi +echo "<-------------------------------->" +sleep 3 +} + + +function startDssProject(){ + SERVER_NAME=dss-framework-project-server + SERVER_IP=$DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP + startApp + + SERVER_NAME=dss-framework-orchestrator-server + SERVER_IP=$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP + startApp + + SERVER_NAME=dss-apiservice-server + SERVER_IP=$DSS_APISERVICE_SERVER_INSTALL_IP + startApp + + SERVER_NAME=dss-datapipe-server + SERVER_IP=$DSS_DATAPIPE_SERVER_INSTALL_IP + startApp + + SERVER_NAME=dss-workflow-server + SERVER_IP=$DSS_WORKFLOW_SERVER_INSTALL_IP + startApp + + SERVER_NAME=dss-flow-execution-server + SERVER_IP=$DSS_FLOW_EXECUTION_SERVER_INSTALL_IP + startApp +} + +function checkDssService(){ + SERVER_NAME=dss-framework-project-server + SERVER_IP=$DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP + checkServer + + SERVER_NAME=dss-framework-orchestrator-server + SERVER_IP=$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP + checkServer + + SERVER_NAME=dss-apiservice-server + SERVER_IP=$DSS_APISERVICE_SERVER_INSTALL_IP + checkServer + + SERVER_NAME=dss-datapipe-server + SERVER_IP=$DSS_DATAPIPE_SERVER_INSTALL_IP + checkServer + + SERVER_NAME=dss-workflow-server + SERVER_IP=$DSS_WORKFLOW_SERVER_INSTALL_IP + checkServer + + SERVER_NAME=dss-flow-execution-server + SERVER_IP=$DSS_FLOW_EXECUTION_SERVER_INSTALL_IP + checkServer +} + + + +startDssProject +checkDssService \ No newline at end of file diff --git a/sbin/dss-stop-all.sh b/sbin/dss-stop-all.sh new file mode 100644 index 000000000..803c5f249 --- /dev/null +++ b/sbin/dss-stop-all.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# +# Copyright 2019 WeBank +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# stop all dss applications +info="We will stop all dss applications, it will take some time, please wait" +echo ${info} + +#Actively load user env + + +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi +local_host="`hostname --fqdn`" +source $DSS_HOME/sbin/common.sh + +function stopApp(){ +echo "<-------------------------------->" +echo "Begin to stop $SERVER_NAME" +SERVER_STOP_CMD="sh $DSS_HOME/sbin/dss-daemon.sh stop $SERVER_NAME" +if test -z "$SERVER_IP" +then + SERVER_IP=$local_host +fi +executeCMD $SERVER_IP "$SERVER_STOP_CMD" +echo "<-------------------------------->" +} + +function stopDssProject(){ + SERVER_NAME=dss-framework-project-server + SERVER_IP=$DSS_FRAMEWORK_PROJECT_SERVER_INSTALL_IP + stopApp + + SERVER_NAME=dss-framework-orchestrator-server + SERVER_IP=$DSS_FRAMEWORK_ORCHESTRATOR_SERVER_INSTALL_IP + stopApp + + SERVER_NAME=dss-apiservice-server + SERVER_IP=$DSS_APISERVICE_SERVER_INSTALL_IP + stopApp + + SERVER_NAME=dss-datapipe-server + SERVER_IP=$DSS_DATAPIPE_SERVER_INSTALL_IP + stopApp + + SERVER_NAME=dss-workflow-server + SERVER_IP=$DSS_WORKFLOW_SERVER_INSTALL_IP + stopApp + + SERVER_NAME=dss-flow-execution-server + SERVER_IP=$DSS_FLOW_EXECUTION_SERVER_INSTALL_IP + stopApp +} + +stopDssProject \ No newline at end of file diff --git a/sbin/ext/dss-apiservice-server b/sbin/ext/dss-apiservice-server new file mode 100644 index 000000000..5921720b6 --- /dev/null +++ b/sbin/ext/dss-apiservice-server @@ -0,0 +1,83 @@ +#!/bin/bash +# +# description: ecm start cmd +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for Linkis 1.0.0 + +# get log directory +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +SERVER_SUFFIX="dss-apiservice-server" +## set log +if [ "$DSS_LOG_DIR" = "" ]; then + export DSS_LOG_DIR="$DSS_HOME/logs" +fi +export SERVER_LOG_PATH=$DSS_LOG_DIR +if [ ! -w "$SERVER_LOG_PATH" ] ; then + mkdir -p "$SERVER_LOG_PATH" +fi + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS="-DserviceName=$SERVER_SUFFIX -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$SERVER_LOG_PATH/$SERVER_SUFFIX.log" +fi + +export SERVER_CLASS=com.webank.wedatasphere.dss.apiservice.DSSApiServiceServerApplication + +## conf dir +export SERVER_CONF_PATH=$DSS_CONF_DIR:$DSS_CONF_DIR/$SERVER_SUFFIX + +## commons lib +export DSS_COMMONS_LIB="$DSS_HOME/lib/dss-commons" + + +if [ ! -r "$DSS_COMMONS_LIB" ] ; then + echo "dss commons lib not exists $DSS_COMMONS_LIB" + exit 1 +fi + +## server lib +export SERVER_LIB=$DSS_HOME/lib/dss-apps/$SERVER_SUFFIX + + +if [ ! -r "$SERVER_LIB" ] ; then + echo "server lib not exists $SERVER_LIB" + exit 1 +fi + +export SERVER_PID=$DSS_HOME/pid/${SERVER_SUFFIX}.pid +## set class path +export SERVER_CLASS_PATH=$SERVER_CONF_PATH:$DSS_COMMONS_LIB/*:$SERVER_LIB/* + +nohup java $SERVER_JAVA_OPTS -cp $SERVER_CLASS_PATH $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/$SERVER_SUFFIX.out & +pid=$! + +sleep 2 +if [[ -z "${pid}" ]]; then + echo "server $SERVER_SUFFIX start failed!" + exit 1 +else + echo "server $SERVER_SUFFIX start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi +exit 1 diff --git a/sbin/ext/dss-datapipe-server b/sbin/ext/dss-datapipe-server new file mode 100644 index 000000000..414859807 --- /dev/null +++ b/sbin/ext/dss-datapipe-server @@ -0,0 +1,83 @@ +#!/bin/bash +# +# description: ecm start cmd +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for Linkis 1.0.0 + +# get log directory +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +SERVER_SUFFIX="dss-datapipe-server" +## set log +if [ "$DSS_LOG_DIR" = "" ]; then + export DSS_LOG_DIR="$DSS_HOME/logs" +fi +export SERVER_LOG_PATH=$DSS_LOG_DIR +if [ ! -w "$SERVER_LOG_PATH" ] ; then + mkdir -p "$SERVER_LOG_PATH" +fi + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS="-DserviceName=$SERVER_SUFFIX -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$SERVER_LOG_PATH/$SERVER_SUFFIX.log" +fi + +export SERVER_CLASS=com.webank.wedatasphere.dss.datapipe.DSSDatapipeServerApplication + +## conf dir +export SERVER_CONF_PATH=$DSS_CONF_DIR:$DSS_CONF_DIR/$SERVER_SUFFIX + +## commons lib +export DSS_COMMONS_LIB="$DSS_HOME/lib/dss-commons" + + +if [ ! -r "$DSS_COMMONS_LIB" ] ; then + echo "dss commons lib not exists $DSS_COMMONS_LIB" + exit 1 +fi + +## server lib +export SERVER_LIB=$DSS_HOME/lib/dss-apps/$SERVER_SUFFIX + + +if [ ! -r "$SERVER_LIB" ] ; then + echo "server lib not exists $SERVER_LIB" + exit 1 +fi + +export SERVER_PID=$DSS_HOME/pid/${SERVER_SUFFIX}.pid +## set class path +export SERVER_CLASS_PATH=$SERVER_CONF_PATH:$DSS_COMMONS_LIB/*:$SERVER_LIB/* + +nohup java $SERVER_JAVA_OPTS -cp $SERVER_CLASS_PATH $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/$SERVER_SUFFIX.out & +pid=$! + +sleep 2 +if [[ -z "${pid}" ]]; then + echo "server $SERVER_SUFFIX start failed!" + exit 1 +else + echo "server $SERVER_SUFFIX start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi +exit 1 diff --git a/sbin/ext/dss-flow-execution-server b/sbin/ext/dss-flow-execution-server new file mode 100644 index 000000000..6b02ddb52 --- /dev/null +++ b/sbin/ext/dss-flow-execution-server @@ -0,0 +1,83 @@ +#!/bin/bash +# +# description: ecm start cmd +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for Linkis 1.0.0 + +# get log directory +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +SERVER_SUFFIX="dss-flow-execution-server" +## set log +if [ "$DSS_LOG_DIR" = "" ]; then + export DSS_LOG_DIR="$DSS_HOME/logs" +fi +export SERVER_LOG_PATH=$DSS_LOG_DIR +if [ ! -w "$SERVER_LOG_PATH" ] ; then + mkdir -p "$SERVER_LOG_PATH" +fi + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS="-DserviceName=$SERVER_SUFFIX -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$SERVER_LOG_PATH/$SERVER_SUFFIX.log" +fi + +export SERVER_CLASS=com.webank.wedatasphere.dss.flow.execution.entrance.DSSFowExecutionServerApplication + +## conf dir +export SERVER_CONF_PATH=$DSS_CONF_DIR:$DSS_CONF_DIR/$SERVER_SUFFIX + +## commons lib +export DSS_COMMONS_LIB="$DSS_HOME/lib/dss-commons" + + +if [ ! -r "$DSS_COMMONS_LIB" ] ; then + echo "dss commons lib not exists $DSS_COMMONS_LIB" + exit 1 +fi + +## server lib +export SERVER_LIB=$DSS_HOME/lib/dss-orchestrator/$SERVER_SUFFIX + + +if [ ! -r "$SERVER_LIB" ] ; then + echo "server lib not exists $SERVER_LIB" + exit 1 +fi + +export SERVER_PID=$DSS_HOME/pid/${SERVER_SUFFIX}.pid +## set class path +export SERVER_CLASS_PATH=$SERVER_CONF_PATH:$DSS_COMMONS_LIB/*:$SERVER_LIB/* + +nohup java $SERVER_JAVA_OPTS -cp $SERVER_CLASS_PATH $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/$SERVER_SUFFIX.out & +pid=$! + +sleep 2 +if [[ -z "${pid}" ]]; then + echo "server $SERVER_SUFFIX start failed!" + exit 1 +else + echo "server $SERVER_SUFFIX start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi +exit 1 diff --git a/sbin/ext/dss-framework-orchestrator-server b/sbin/ext/dss-framework-orchestrator-server new file mode 100644 index 000000000..d112e8009 --- /dev/null +++ b/sbin/ext/dss-framework-orchestrator-server @@ -0,0 +1,83 @@ +#!/bin/bash +# +# description: ecm start cmd +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for Linkis 1.0.0 + +# get log directory +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +SERVER_SUFFIX="dss-framework-orchestrator-server" +## set log +if [ "$DSS_LOG_DIR" = "" ]; then + export DSS_LOG_DIR="$DSS_HOME/logs" +fi +export SERVER_LOG_PATH=$DSS_LOG_DIR +if [ ! -w "$SERVER_LOG_PATH" ] ; then + mkdir -p "$SERVER_LOG_PATH" +fi + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS="-DserviceName=$SERVER_SUFFIX -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$SERVER_LOG_PATH/$SERVER_SUFFIX.log" +fi + +export SERVER_CLASS=com.webank.wedatasphere.dss.orchestrator.server.DSSOrchestratorServerApplication + +## conf dir +export SERVER_CONF_PATH=$DSS_CONF_DIR:$DSS_CONF_DIR/$SERVER_SUFFIX + +## commons lib +export DSS_COMMONS_LIB="$DSS_HOME/lib/dss-commons" + + +if [ ! -r "$DSS_COMMONS_LIB" ] ; then + echo "dss commons lib not exists $DSS_COMMONS_LIB" + exit 1 +fi + +## server lib +export SERVER_LIB=$DSS_HOME/lib/dss-framework/$SERVER_SUFFIX + + +if [ ! -r "$SERVER_LIB" ] ; then + echo "server lib not exists $SERVER_LIB" + exit 1 +fi + +export SERVER_PID=$DSS_HOME/pid/${SERVER_SUFFIX}.pid +## set class path +export SERVER_CLASS_PATH=$SERVER_CONF_PATH:$DSS_COMMONS_LIB/*:$SERVER_LIB/* + +nohup java $SERVER_JAVA_OPTS -cp $SERVER_CLASS_PATH $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/$SERVER_SUFFIX.out & +pid=$! + +sleep 2 +if [[ -z "${pid}" ]]; then + echo "server $SERVER_SUFFIX start failed!" + exit 1 +else + echo "server $SERVER_SUFFIX start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi +exit 1 diff --git a/sbin/ext/dss-framework-project-server b/sbin/ext/dss-framework-project-server new file mode 100644 index 000000000..50624981d --- /dev/null +++ b/sbin/ext/dss-framework-project-server @@ -0,0 +1,83 @@ +#!/bin/bash +# +# description: ecm start cmd +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for Linkis 1.0.0 + +# get log directory +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +SERVER_SUFFIX="dss-framework-project-server" +## set log +if [ "$DSS_LOG_DIR" = "" ]; then + export DSS_LOG_DIR="$DSS_HOME/logs" +fi +export SERVER_LOG_PATH=$DSS_LOG_DIR +if [ ! -w "$SERVER_LOG_PATH" ] ; then + mkdir -p "$SERVER_LOG_PATH" +fi + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS="-DserviceName=$SERVER_SUFFIX -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$SERVER_LOG_PATH/$SERVER_SUFFIX.log" +fi + +export SERVER_CLASS=com.webank.wedatasphere.dss.framework.project.server.DSSProjectServerApplication + +## conf dir +export SERVER_CONF_PATH=$DSS_CONF_DIR:$DSS_CONF_DIR/$SERVER_SUFFIX + +## commons lib +export DSS_COMMONS_LIB="$DSS_HOME/lib/dss-commons" + + +if [ ! -r "$DSS_COMMONS_LIB" ] ; then + echo "dss commons lib not exists $DSS_COMMONS_LIB" + exit 1 +fi + +## server lib +export SERVER_LIB=$DSS_HOME/lib/dss-framework/$SERVER_SUFFIX + + +if [ ! -r "$SERVER_LIB" ] ; then + echo "server lib not exists $SERVER_LIB" + exit 1 +fi + +export SERVER_PID=$DSS_HOME/pid/${SERVER_SUFFIX}.pid +## set class path +export SERVER_CLASS_PATH=$SERVER_CONF_PATH:$DSS_COMMONS_LIB/*:$SERVER_LIB/* + +nohup java $SERVER_JAVA_OPTS -cp $SERVER_CLASS_PATH $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/$SERVER_SUFFIX.out & +pid=$! + +sleep 2 +if [[ -z "${pid}" ]]; then + echo "server $SERVER_SUFFIX start failed!" + exit 1 +else + echo "server $SERVER_SUFFIX start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi +exit 1 diff --git a/sbin/ext/dss-workflow-server b/sbin/ext/dss-workflow-server new file mode 100644 index 000000000..f410ec944 --- /dev/null +++ b/sbin/ext/dss-workflow-server @@ -0,0 +1,83 @@ +#!/bin/bash +# +# description: ecm start cmd +# @author: peacewong +# @created: 01.16.2021 +# +# Modified for Linkis 1.0.0 + +# get log directory +cd `dirname $0` +cd .. +INSTALL_HOME=`pwd` + +# set DSS_HOME +if [ "$DSS_HOME" = "" ]; then + export DSS_HOME=$INSTALL_HOME +fi + +# set DSS_CONF_DIR +if [ "$DSS_CONF_DIR" = "" ]; then + export DSS_CONF_DIR=$DSS_HOME/conf +fi + +SERVER_SUFFIX="dss-workflow-server" +## set log +if [ "$DSS_LOG_DIR" = "" ]; then + export DSS_LOG_DIR="$DSS_HOME/logs" +fi +export SERVER_LOG_PATH=$DSS_LOG_DIR +if [ ! -w "$SERVER_LOG_PATH" ] ; then + mkdir -p "$SERVER_LOG_PATH" +fi + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS="-DserviceName=$SERVER_SUFFIX -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$SERVER_LOG_PATH/$SERVER_SUFFIX.log" +fi + +export SERVER_CLASS=com.webank.wedatasphere.dss.workflow.DSSWorkflowServerApplication + +## conf dir +export SERVER_CONF_PATH=$DSS_CONF_DIR:$DSS_CONF_DIR/$SERVER_SUFFIX + +## commons lib +export DSS_COMMONS_LIB="$DSS_HOME/lib/dss-commons" + + +if [ ! -r "$DSS_COMMONS_LIB" ] ; then + echo "dss commons lib not exists $DSS_COMMONS_LIB" + exit 1 +fi + +## server lib +export SERVER_LIB=$DSS_HOME/lib/dss-orchestrator/$SERVER_SUFFIX + + +if [ ! -r "$SERVER_LIB" ] ; then + echo "server lib not exists $SERVER_LIB" + exit 1 +fi + +export SERVER_PID=$DSS_HOME/pid/${SERVER_SUFFIX}.pid +## set class path +export SERVER_CLASS_PATH=$SERVER_CONF_PATH:$DSS_COMMONS_LIB/*:$SERVER_LIB/* + +nohup java $SERVER_JAVA_OPTS -cp $SERVER_CLASS_PATH $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/$SERVER_SUFFIX.out & +pid=$! + +sleep 2 +if [[ -z "${pid}" ]]; then + echo "server $SERVER_SUFFIX start failed!" + exit 1 +else + echo "server $SERVER_SUFFIX start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi +exit 1 diff --git a/sendemail-appjoint/sendemail-core/pom.xml b/sendemail-appjoint/sendemail-core/pom.xml deleted file mode 100644 index 3739647f0..000000000 --- a/sendemail-appjoint/sendemail-core/pom.xml +++ /dev/null @@ -1,129 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-sendmail-appjoint-core - - - - com.webank.wedatasphere.dss - dss-appjoint-core - ${dss.version} - provided - true - - - - com.webank.wedatasphere.linkis - linkis-module - ${linkis.version} - provided - true - - - - org.springframework - spring-context-support - 5.0.7.RELEASE - - - - javax.mail - mail - 1.4 - - - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - src/main/java - - **/*.xml - - - - src/main/resources - - **/*.properties - **/application.yml - **/bootstrap.yml - **/log4j2.xml - - - - - - - \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointSpringConfiguration.java b/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointSpringConfiguration.java deleted file mode 100644 index 856c29233..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointSpringConfiguration.java +++ /dev/null @@ -1,80 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.conf; - -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContentGenerator; -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailGenerator; -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailSender; -import com.webank.wedatasphere.dss.appjoint.sendemail.email.generate.MultiContentEmailGenerator; -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.generator.MultiEmailContentGenerator; -import com.webank.wedatasphere.dss.appjoint.sendemail.sender.SpringJavaEmailSender; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Created by enjoyyin on 2019/10/14. - */ - -public class SendEmailAppJointSpringConfiguration { - - - private static final String ESB_EMAIL_SENDER = "com.webank.wedatasphere.dss.appjoint.sendemail.sender.EsbEmailSender"; - - private static final Logger logger = LoggerFactory.getLogger(SendEmailAppJointSpringConfiguration.class); - - private static final EmailGenerator EMAIL_GENERATOR = new MultiContentEmailGenerator(); - - private static final EmailSender SPRING_EMAIL_SENDER = new SpringJavaEmailSender(); - - private static final EmailContentGenerator EMAIL_CONTENT_GENERATOR = new MultiEmailContentGenerator(); - - private static EmailSender emailSender = null; - - static { - try { - Class clazz = Class.forName(ESB_EMAIL_SENDER); - emailSender = (EmailSender) clazz.newInstance(); - } catch (Exception e) { - logger.warn("{} can be instanced", ESB_EMAIL_SENDER, e); - } - } - /** - * 如果是行内就直接返回com.webank.wedatasphere.dss.appjoint.sendemail.sender.EsbEmailSender - * @return - */ - - public static EmailSender createEmailSender() { - - if (null != emailSender) { - return emailSender; - } else{ - return SPRING_EMAIL_SENDER; - } - } - - - public static EmailGenerator createEmailGenerator() { - return EMAIL_GENERATOR; - } - - - public static EmailContentGenerator createEmailContentGenerator() { - return EMAIL_CONTENT_GENERATOR; - } - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/sender/AbstractEmailSender.java b/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/sender/AbstractEmailSender.java deleted file mode 100644 index 29536a1e6..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/sender/AbstractEmailSender.java +++ /dev/null @@ -1,40 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.sender; - -import com.webank.wedatasphere.dss.appjoint.sendemail.Email; -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailSender; -import com.webank.wedatasphere.linkis.common.utils.Utils; -import scala.runtime.BoxedUnit; - -import java.util.concurrent.Future; - -/** - * Created by enjoyyin on 2019/10/13. - */ -public abstract class AbstractEmailSender implements EmailSender { - - @Override - public Future sendAsync(Email email) { - return Utils.defaultScheduler().submit(() -> { - send(email); - return BoxedUnit.UNIT; - }); - } - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/sender/SpringJavaEmailSender.java b/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/sender/SpringJavaEmailSender.java deleted file mode 100644 index eb7a6c56d..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/java/com/webank/wedatasphere/dss/appjoint/sendemail/sender/SpringJavaEmailSender.java +++ /dev/null @@ -1,99 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.sender; - -import com.webank.wedatasphere.dss.appjoint.sendemail.conf.SendEmailAppJointConfiguration; -import com.webank.wedatasphere.dss.appjoint.sendemail.Attachment; -import com.webank.wedatasphere.dss.appjoint.sendemail.Email; -import com.webank.wedatasphere.dss.appjoint.sendemail.exception.EmailSendFailedException; -import org.apache.commons.lang.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.mail.javamail.JavaMailSender; -import org.springframework.mail.javamail.JavaMailSenderImpl; -import org.springframework.mail.javamail.MimeMessageHelper; - -import javax.mail.internet.MimeMessage; -import javax.mail.util.ByteArrayDataSource; -import java.util.Properties; - -/** - * Created by shanhuang on 2019/10/12. - */ -public class SpringJavaEmailSender extends AbstractEmailSender { - - private static final Logger logger = LoggerFactory.getLogger(SpringJavaEmailSender.class); - - private JavaMailSenderImpl javaMailSender = new JavaMailSenderImpl(); - - public SpringJavaEmailSender() { - javaMailSender.setHost(SendEmailAppJointConfiguration.EMAIL_HOST().getValue()); - javaMailSender.setPort(Integer.parseInt(SendEmailAppJointConfiguration.EMAIL_PORT().getValue())); - javaMailSender.setProtocol(SendEmailAppJointConfiguration.EMAIL_PROTOCOL().getValue()); - javaMailSender.setUsername(SendEmailAppJointConfiguration.EMAIL_USERNAME().getValue()); - javaMailSender.setPassword(SendEmailAppJointConfiguration.EMAIL_PASSWORD().getValue()); - try { - Properties prop = new Properties(); - prop.put("mail.smtp.auth", Boolean.parseBoolean(SendEmailAppJointConfiguration.EMAIL_SMTP_AUTH().getValue())); - prop.put("mail.smtp.starttls.enable", Boolean.parseBoolean(SendEmailAppJointConfiguration.EMAIL_SMTP_STARTTLS_ENABLE().getValue())); - prop.put("mail.smtp.starttls.required", Boolean.parseBoolean(SendEmailAppJointConfiguration.EMAIL_SMTP_STARTTLS_REQUIRED().getValue())); - prop.put("mail.smtp.ssl.enable", Boolean.parseBoolean(SendEmailAppJointConfiguration.EMAIL_SMTP_SSL_ENABLED().getValue())); - prop.put("mail.smtp.timeout", Integer.parseInt(SendEmailAppJointConfiguration.EMAIL_SMTP_TIMEOUT().getValue())); - javaMailSender.setJavaMailProperties(prop); - } catch (Exception e) { - logger.error("Failed to read mail properties, roll back to default values.", e); - } - } - - @Override - public void send(Email email) throws EmailSendFailedException { - logger.info("Begin to send Email(" + email.getSubject() + ")."); - try { - javaMailSender.send(parseToMimeMessage(email)); - } catch (Exception e) { - logger.error("Send email failed: ", e); - EmailSendFailedException ex = new EmailSendFailedException(80001, "Send email failed!"); - ex.initCause(e); - throw ex; - } - logger.info("Send Email(" + email.getSubject() + ") succeed."); - } - - private MimeMessage parseToMimeMessage(Email email) { - MimeMessage message = javaMailSender.createMimeMessage(); - try { - MimeMessageHelper messageHelper = new MimeMessageHelper(message, true); - if(StringUtils.isBlank(email.getFrom())) { - messageHelper.setFrom(SendEmailAppJointConfiguration.DEFAULT_EMAIL_FROM().getValue()); - } else { - messageHelper.setFrom(email.getFrom()); - } - messageHelper.setSubject(email.getSubject()); - messageHelper.setTo(email.getTo()); - messageHelper.setCc(email.getCc()); - messageHelper.setBcc(email.getBcc()); - for(Attachment attachment: email.getAttachments()){ - messageHelper.addAttachment(attachment.getName(), new ByteArrayDataSource(attachment.getBase64Str(), attachment.getMediaType())); - } - messageHelper.setText(email.getContent(), true); - } catch (Exception e) { - logger.error("Send mail failed", e); - } - return message; - } -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContentGenerator.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContentGenerator.scala deleted file mode 100644 index db6e7ccc0..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContentGenerator.scala +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -/** - * Created by shanhuang on 2019/10/12. - */ -trait EmailContentGenerator { - - def generate(email: Email): Unit - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContentParser.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContentParser.scala deleted file mode 100644 index 19b790146..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailContentParser.scala +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -/** - * Created by shanhuang on 2019/10/12. - */ -trait EmailContentParser { - - def parse(email: Email): Unit - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailGenerator.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailGenerator.scala deleted file mode 100644 index e6cefee32..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailGenerator.scala +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, NodeContext} - -/** - * Created by enjoyyin on 2019/10/12. - */ -trait EmailGenerator { - - def generateEmail(node: AppJointNode, nodeContext: NodeContext): Email - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailSender.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailSender.scala deleted file mode 100644 index 949fb19ee..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/EmailSender.scala +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -import java.util.concurrent.Future - -import com.webank.wedatasphere.dss.appjoint.sendemail.exception.EmailSendFailedException -import com.webank.wedatasphere.dss.appjoint.sendemail.exception.EmailSendFailedException - -/** - * Created by shanhuang on 2019/10/12. - */ -trait EmailSender { - - @throws(classOf[EmailSendFailedException]) - def send(email: Email): Unit - - def sendAsync(email: Email): Future[Unit] - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailAppJoint.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailAppJoint.scala deleted file mode 100644 index 97235acf9..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailAppJoint.scala +++ /dev/null @@ -1,51 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -import java.util - -import com.webank.wedatasphere.dss.appjoint.AppJoint -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl - -/** - * Created by shanhuang on 2019/10/12. - */ -class SendEmailAppJoint extends AppJointUrlImpl with AppJoint { - - private var params: util.Map[String, AnyRef] = _ - private var nodeExecution: NodeExecution = _ - - override def getAppJointName = "SendEmail" - - override def init(baseUrl: String, params: util.Map[String, AnyRef]): Unit = { - setBaseUrl(baseUrl) - this.params = params - } - - override def getNodeExecution: NodeExecution = { - if(nodeExecution == null) synchronized { - if(nodeExecution == null) { - nodeExecution = new SendEmailNodeExecution - nodeExecution.setBaseUrl(getBaseUrl) - nodeExecution.init(params) - } - } - nodeExecution - } -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailNodeExecution.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailNodeExecution.scala deleted file mode 100644 index 9f81f3724..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailNodeExecution.scala +++ /dev/null @@ -1,85 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -import java.util - -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution -import com.webank.wedatasphere.dss.appjoint.execution.common.{CompletedNodeExecutionResponse, NodeExecutionResponse} -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, NodeContext} -import com.webank.wedatasphere.dss.appjoint.sendemail.conf.SendEmailAppJointSpringConfiguration -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.parser.{FileEmailContentParser, HtmlEmailContentParser, PictureEmailContentParser, TableEmailContentParser} -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl -import com.webank.wedatasphere.dss.appjoint.service.session.Session -import com.webank.wedatasphere.linkis.common.exception.ErrorException -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} - -/** - * Created by shanhuang on 2019/10/12. - */ -class SendEmailNodeExecution extends AppJointUrlImpl with NodeExecution with Logging{ - - private val sendEmailAppJointHooks = Array.empty[SendEmailNodeExecutionHook] - private val emailContentParsers = - Array[EmailContentParser](FileEmailContentParser, HtmlEmailContentParser, PictureEmailContentParser, TableEmailContentParser) - private val emailContentGenerators = Array[EmailContentGenerator](SendEmailAppJointSpringConfiguration.createEmailContentGenerator()) - - private val emailGenerator = SendEmailAppJointSpringConfiguration.createEmailGenerator() - - private val emailSender = SendEmailAppJointSpringConfiguration.createEmailSender() - - override def canExecute(node: AppJointNode, nodeContext: NodeContext, session: Session): Boolean = node.getNodeType.toLowerCase.contains("sendemail") - - override def execute(node: AppJointNode, nodeContext: NodeContext, seesion:Session): NodeExecutionResponse = { - val response = new CompletedNodeExecutionResponse - val email = Utils.tryCatch { - sendEmailAppJointHooks.foreach(_.preGenerate(node, nodeContext)) - val email = emailGenerator.generateEmail(node, nodeContext) - emailContentParsers.foreach{ - p => Utils.tryQuietly(p.parse(email)) - } - emailContentGenerators.foreach{ - g => Utils.tryQuietly(g.generate(email)) - } - sendEmailAppJointHooks.foreach(_.preSend(node, nodeContext, email)) - email - }{ t => - putErrorMsg("解析邮件内容失败！", t, response) - return response - } - Utils.tryCatch { - emailSender.send(email) - response.setIsSucceed(true) - }(putErrorMsg("发送邮件失败！", _, response)) - response - } - - protected def putErrorMsg(errorMsg: String, t: Throwable, - response: CompletedNodeExecutionResponse): Unit = t match { - case t: Throwable => - response.setErrorMsg(errorMsg) - val exception = new ErrorException(80079, "failed to sendEmail") - exception.initCause(t) - logger.error(s"failed to send email, $errorMsg ", t) - response.setException(exception) - response.setIsSucceed(false) - } - - - override def init(params: util.Map[String, AnyRef]): Unit = {} -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailNodeExecutionHook.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailNodeExecutionHook.scala deleted file mode 100644 index e13371be2..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/SendEmailNodeExecutionHook.scala +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail - -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, NodeContext} - -/** - * Created by enjoyyin on 2019/10/12. - */ -trait SendEmailNodeExecutionHook { - - def preGenerate(node: AppJointNode, nodeContext: NodeContext): Unit - - def preSend(node: AppJointNode, nodeContext: NodeContext, email: Email): Unit - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointConfiguration.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointConfiguration.scala deleted file mode 100644 index 1f271c95b..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointConfiguration.scala +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.conf - -import com.webank.wedatasphere.linkis.common.conf.CommonVars - -/** - * Created by shanhuang on 2019/10/12. - */ -object SendEmailAppJointConfiguration { - - val EMAIL_IMAGE_HEIGHT = CommonVars("wds.dataworkis.appjoint.email.image.height", 500) - val EMAIL_IMAGE_WIDTH = CommonVars("wds.dataworkis.appjoint.email.image.width", 1920) - val DEFAULT_EMAIL_FROM = CommonVars("wds.dataworkis.appjoint.email.from.default", "") - - val EMAIL_HOST = CommonVars("wds.dataworkis.appjoint.email.host", "") - val EMAIL_PORT = CommonVars("wds.dataworkis.appjoint.email.port", "") - val EMAIL_PROTOCOL = CommonVars("wds.dataworkis.appjoint.email.protocol", "smtp") - val EMAIL_USERNAME = CommonVars("wds.dataworkis.appjoint.email.username", "") - val EMAIL_PASSWORD = CommonVars("wds.dataworkis.appjoint.email.password", "") - - val EMAIL_SMTP_AUTH = CommonVars("wds.dataworkis.appjoint.email.smtp.auth", "true") - val EMAIL_SMTP_STARTTLS_ENABLE = CommonVars("wds.dataworkis.appjoint.email.smtp.starttls.enable", "true") - val EMAIL_SMTP_STARTTLS_REQUIRED = CommonVars("wds.dataworkis.appjoint.email.smtp.starttls.required", "true") - val EMAIL_SMTP_SSL_ENABLED = CommonVars("wds.dataworkis.appjoint.email.smtp.ssl.enable", "true") - val EMAIL_SMTP_TIMEOUT = CommonVars("wds.dataworkis.appjoint.email.smtp.timeout", "25000") - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointUtils.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointUtils.scala deleted file mode 100644 index 38ca58210..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/conf/SendEmailAppJointUtils.scala +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.conf - -/** - * Created by enjoyyin on 2019/10/12. - */ -object SendEmailAppJointUtils { - -// def getInstance[T](clazz: Class[T]): T = DataWorkCloudApplication.getApplicationContext.getBean(clazz) -// -// def getInstances[T](clazz: Class[T]): Seq[T] = { -// val instances = ArrayBuffer[T]() -// DataWorkCloudApplication.getApplicationContext.getBeansOfType(clazz).foreach(instances += _._2) -// instances -// } - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/MultiContentEmail.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/MultiContentEmail.scala deleted file mode 100644 index 4fde2ac36..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/MultiContentEmail.scala +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.email - -import java.util - -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContent -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContent - -import scala.collection.JavaConversions._ - -/** - * Created by shanhuang on 2019/10/12. - */ -class MultiContentEmail extends AbstractEmail { - - private val emailContents = new util.ArrayList[EmailContent[_]]() - - def addEmailContent(emailContent: EmailContent[_]): Unit = emailContents.add(emailContent) - - def getEmailContents: Array[EmailContent[_]] = emailContents.toIterator.toArray - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/PngAttachment.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/PngAttachment.scala deleted file mode 100644 index 94c6e12cc..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/PngAttachment.scala +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.email - -import java.io.File - -import com.webank.wedatasphere.dss.appjoint.sendemail.Attachment -import com.webank.wedatasphere.dss.appjoint.sendemail.Attachment - -/** - * Created by shanhuang on 2019/10/12. - */ -class PngAttachment(name: String, b64: String) extends Attachment { - - override def getName: String = name - - override def getBase64Str: String = b64 - - override def getFile: File = null //TODO write b64 to file - - override def getMediaType = "image/png" - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/generate/AbstractEmailGenerator.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/generate/AbstractEmailGenerator.scala deleted file mode 100644 index 2cc77339e..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/generate/AbstractEmailGenerator.scala +++ /dev/null @@ -1,61 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.email.generate - -import com.webank.wedatasphere.dss.appjoint.sendemail.email.AbstractEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.Email -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, NodeContext} -import com.webank.wedatasphere.dss.appjoint.sendemail.email.AbstractEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.{Email, EmailGenerator} -import com.webank.wedatasphere.linkis.common.utils.Logging - -/** - * Created by shanhuang on 2019/10/12. - */ -trait AbstractEmailGenerator extends EmailGenerator with Logging{ - - protected def createEmail(): AbstractEmail - - override def generateEmail(node: AppJointNode, nodeContext: NodeContext): Email = { - val email = createEmail() - generateEmailInfo(node, nodeContext, email) - generateEmailContent(node, nodeContext, email) - email - } - - protected def generateEmailInfo(node: AppJointNode, nodeContext: NodeContext, email: AbstractEmail): Unit = { - import scala.collection.JavaConversions._ - nodeContext.getRuntimeMap foreach { - case (k, v) => logger.info(s"K is $k, V is $v") - } - val subject = if (nodeContext.getRuntimeMap.get("subject") != null) nodeContext.getRuntimeMap.get("subject").toString else "This is an email" - email.setSubject(subject) - val bcc = if (nodeContext.getRuntimeMap.get("bcc") != null) nodeContext.getRuntimeMap.get("bcc").toString else "" - email.setBcc(bcc) - val cc = if (nodeContext.getRuntimeMap.get("cc") != null) nodeContext.getRuntimeMap.get("cc").toString else "" - email.setCc(cc) - val from = if (nodeContext.getRuntimeMap.get("from") != null) nodeContext.getRuntimeMap.get("from").toString else - nodeContext.getRuntimeMap.get("user").toString - email.setFrom(from) - val to = if (nodeContext.getRuntimeMap.get("to") != null) nodeContext.getRuntimeMap.get("to").toString else "" - email.setTo(to) - } - - protected def generateEmailContent(node: AppJointNode, nodeContext: NodeContext, email: AbstractEmail): Unit - -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/generate/MultiContentEmailGenerator.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/generate/MultiContentEmailGenerator.scala deleted file mode 100644 index 3425109e7..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/email/generate/MultiContentEmailGenerator.scala +++ /dev/null @@ -1,63 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.email.generate - -import com.webank.wedatasphere.dss.appjoint.sendemail._ -import com.webank.wedatasphere.dss.appjoint.sendemail.email.{AbstractEmail, MultiContentEmail} -import com.webank.wedatasphere.linkis.common.io.FsPath -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, NodeContext} -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContent -import com.webank.wedatasphere.dss.appjoint.sendemail.email.{AbstractEmail, MultiContentEmail} -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent._ -import com.webank.wedatasphere.linkis.storage.resultset.ResultSetFactory - -/** - * Created by shanhuang on 2019/10/12. - */ -class MultiContentEmailGenerator extends AbstractEmailGenerator { - - override protected def createEmail() = new MultiContentEmail - - override protected def generateEmailContent(node: AppJointNode, nodeContext: NodeContext, email: AbstractEmail): Unit = email match { - case multiContentEmail: MultiContentEmail => - def addContentEmail(getEmailContent: String => EmailContent[_]): Unit = nodeContext.getRuntimeMap.get("content") match { - case content: Array[String] => content.foreach(c => multiContentEmail.addEmailContent(getEmailContent(c))) - case content: String => multiContentEmail.addEmailContent(getEmailContent(content)) - } - nodeContext.getRuntimeMap.get("category") match { - case "node" => - val resultSetFactory = ResultSetFactory.getInstance - nodeContext.getJobIdsOfShareNode.foreach { jobId => - nodeContext.getResultSetPathsByJobId(jobId).foreach { fsPath => - val resultSet = resultSetFactory.getResultSetByPath(fsPath) - val emailContent = resultSet.resultSetType() match { - case ResultSetFactory.PICTURE_TYPE => new PictureEmailContent(fsPath) - case ResultSetFactory.HTML_TYPE => new HtmlEmailContent(fsPath) - case ResultSetFactory.TABLE_TYPE => new TableEmailContent(fsPath) - case ResultSetFactory.TEXT_TYPE => new FileEmailContent(fsPath) - } - multiContentEmail.addEmailContent(emailContent) - } - } - case "file" => addContentEmail(c => new FileEmailContent(new FsPath(c))) - case "text" => addContentEmail(new TextEmailContent(_)) - case "link" => addContentEmail(new UrlEmailContent(_)) - } - } - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/ArrayEmailContent.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/ArrayEmailContent.scala deleted file mode 100644 index 2b1382825..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/ArrayEmailContent.scala +++ /dev/null @@ -1,34 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent - -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContent -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContent - -/** - * Created by enjoyyin on 2019/10/13. - */ -class ArrayEmailContent extends EmailContent[Array[String]] { - - private var content: Array[String] = _ - - override def getContent: Array[String] = content - - override def setContent(content: Array[String]): Unit = this.content = content - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/StringEmailContent.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/StringEmailContent.scala deleted file mode 100644 index c3065f7ba..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/StringEmailContent.scala +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent - -import com.webank.wedatasphere.dss.appjoint.sendemail.EmailContent - -/** - * Created by enjoyyin on 2019/10/12. - */ -class StringEmailContent extends EmailContent[String] { - private var content: String = _ - - override def getContent: String = content - - override def setContent(content: String): Unit = this.content = content - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/generator/MultiEmailContentGenerator.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/generator/MultiEmailContentGenerator.scala deleted file mode 100644 index c48e0115c..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/generator/MultiEmailContentGenerator.scala +++ /dev/null @@ -1,56 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.generator - -import com.webank.wedatasphere.dss.appjoint.sendemail.Email -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.{ArrayEmailContent, StringEmailContent} -import com.webank.wedatasphere.dss.appjoint.sendemail.Email -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.{ArrayEmailContent, StringEmailContent} -import com.webank.wedatasphere.linkis.common.utils.Logging -import org.springframework.stereotype.Component - -/** - * Created by shanhuang on 2019/10/12. - */ - -class MultiEmailContentGenerator extends AbstractEmailContentGenerator with Logging { - - override def generate(email: Email): Unit = email match { - case multiContentEmail: MultiContentEmail => - formatSubjectOfOldVersion(email) - formatSubject(multiContentEmail) - formatContent(multiContentEmail) - } - - protected def formatContent(email: MultiContentEmail): Unit = { - val sb = new StringBuilder("") - sb.append("") - email.getEmailContents.foreach { - case emailContent: ArrayEmailContent => - emailContent.getContent.foreach(content => sb.append("")) - case emailContent: StringEmailContent => - sb.append("") - } - sb.append("
").append(content).append("
").append(emailContent.getContent).append("
") - sb.append("") - email.setContent(sb.toString) - } - -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/FileEmailContentParser.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/FileEmailContentParser.scala deleted file mode 100644 index 7181342c8..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/FileEmailContentParser.scala +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.parser - -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.FileEmailContent -import com.webank.wedatasphere.linkis.common.utils.Utils -import com.webank.wedatasphere.linkis.storage.LineRecord -import org.apache.commons.io.IOUtils -import org.springframework.stereotype.Component - -/** - * Created by shanhuang on 2019/10/12. - */ - -object FileEmailContentParser extends AbstractEmailContentParser[FileEmailContent] { - override protected def parseEmailContent(emailContent: FileEmailContent, - multiContentEmail: MultiContentEmail): Unit = { - val reader = getResultSetReader(emailContent) - val content = new StringBuilder - Utils.tryFinally{ - while(reader.hasNext) reader.getRecord match { - case lineRecord: LineRecord => - content.append(lineRecord.getLine).append("
") - } - }(IOUtils.closeQuietly(reader)) - emailContent.setContent(content.toString()) - } -} \ No newline at end of file diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/HtmlEmailContentParser.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/HtmlEmailContentParser.scala deleted file mode 100644 index 71ef0d0ec..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/HtmlEmailContentParser.scala +++ /dev/null @@ -1,33 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.parser - -import com.webank.wedatasphere.dss.appjoint.sendemail.email.MultiContentEmail -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.HtmlEmailContent -import org.springframework.stereotype.Component - -/** - * Created by shanhuang on 2019/10/12. - */ - -object HtmlEmailContentParser extends AbstractEmailContentParser[HtmlEmailContent] { - override protected def parseEmailContent(emailContent: HtmlEmailContent, - multiContentEmail: MultiContentEmail): Unit = { - getFirstLineRecord(emailContent).foreach(emailContent.setContent) - } -} diff --git a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/PictureEmailContentParser.scala b/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/PictureEmailContentParser.scala deleted file mode 100644 index 595c417ac..000000000 --- a/sendemail-appjoint/sendemail-core/src/main/scala/com/webank/wedatasphere/dss/appjoint/sendemail/emailcontent/parser/PictureEmailContentParser.scala +++ /dev/null @@ -1,74 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.parser - -import java.awt.image.BufferedImage -import java.io.{ByteArrayInputStream, ByteArrayOutputStream, File} -import java.util.{Base64, UUID} - -import javax.imageio.ImageIO -import com.webank.wedatasphere.dss.appjoint.sendemail.conf.SendEmailAppJointConfiguration.EMAIL_IMAGE_HEIGHT -import com.webank.wedatasphere.dss.appjoint.sendemail.email.{AbstractEmail, MultiContentEmail, PngAttachment} -import com.webank.wedatasphere.dss.appjoint.sendemail.emailcontent.PictureEmailContent -import com.webank.wedatasphere.linkis.common.conf.Configuration -import org.apache.commons.codec.binary.Base64OutputStream -import org.springframework.stereotype.Component - -/** - * Created by shanhuang on 2019/10/12. - */ - -object PictureEmailContentParser extends AbstractEmailContentParser[PictureEmailContent] { - - override protected def parseEmailContent(emailContent: PictureEmailContent, - multiContentEmail: MultiContentEmail): Unit = { - getFirstLineRecord(emailContent).foreach { imageStr => - val decoder = Base64.getDecoder - val byteArr = decoder.decode(imageStr) - val inputStream = new ByteArrayInputStream(byteArr) - val image = ImageIO.read(inputStream) - val contents = generateImage(image, multiContentEmail) - emailContent.setContent(contents) -// val image = ImageIO.read(new ByteArrayInputStream(imageStr.getBytes(Configuration.BDP_ENCODING.getValue))) -// val contents = generateImage(image, multiContentEmail) -// emailContent.setContent(contents) - } - } - - protected def generateImage(bufferedImage: BufferedImage, email: AbstractEmail): Array[String] = { - val imageUUID = UUID.randomUUID.toString - val width = bufferedImage.getWidth - val height = bufferedImage.getHeight - val imagesCuts = if (height > EMAIL_IMAGE_HEIGHT.getValue) { - val numOfCut = Math.ceil(height.toDouble / EMAIL_IMAGE_HEIGHT.getValue).toInt - val realHeight = height / numOfCut - (0 until numOfCut).map(i => bufferedImage.getSubimage(0, i * realHeight, width, realHeight)).toArray - } else Array(bufferedImage) - imagesCuts.indices.map { index => - val image = imagesCuts(index) - val imageName = index + "_" + imageUUID - val os = new ByteArrayOutputStream - val b64Stream = new Base64OutputStream(os) - ImageIO.write(image, "png", b64Stream) - val b64 = os.toString(Configuration.BDP_ENCODING.getValue) - email.addAttachment(new PngAttachment(imageName, b64)) - s"""""" - }.toArray - } - -} diff --git a/visualis-appjoint/appjoint/pom.xml b/visualis-appjoint/appjoint/pom.xml deleted file mode 100644 index e572638da..000000000 --- a/visualis-appjoint/appjoint/pom.xml +++ /dev/null @@ -1,130 +0,0 @@ - - - - - - dss - com.webank.wedatasphere.dss - 0.9.1 - - 4.0.0 - - dss-visualis-appjoint - - - - com.webank.wedatasphere.dss - dss-appjoint-core - ${dss.version} - provided - true - - - - com.webank.wedatasphere.linkis - linkis-module - ${linkis.version} - provided - true - - - - com.webank.wedatasphere.linkis - linkis-httpclient - ${linkis.version} - provided - true - - - - net.databinder.dispatch - dispatch-core_2.11 - 0.12.3 - - - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - - - net.alchim31.maven - scala-maven-plugin - - - org.apache.maven.plugins - maven-jar-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - 2.3 - false - - - make-assembly - package - - single - - - - src/main/assembly/distribution.xml - - - - - - false - dss-visualis-appjoint - false - false - - src/main/assembly/distribution.xml - - - - - - - src/main/java - - **/*.xml - - - - src/main/resources - - **/*.properties - **/application.yml - **/bootstrap.yml - **/log4j2.xml - - - - - - - \ No newline at end of file diff --git a/visualis-appjoint/appjoint/src/main/assembly/distribution.xml b/visualis-appjoint/appjoint/src/main/assembly/distribution.xml deleted file mode 100644 index 0d56a79cf..000000000 --- a/visualis-appjoint/appjoint/src/main/assembly/distribution.xml +++ /dev/null @@ -1,226 +0,0 @@ - - - - dss-visualis-appjoint - - zip - - true - visualis - - - - - - lib - true - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ${basedir}/conf - - * - - 0777 - conf - unix - - - ${basedir}/bin - - * - - 0777 - bin - unix - - - . - - */** - - logs - - - - - diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/VisualisAppJoint.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/VisualisAppJoint.java deleted file mode 100644 index 4dff6a3f1..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/VisualisAppJoint.java +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis; - -import com.webank.wedatasphere.dss.appjoint.AppJoint; -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.appjoint.service.NodeService; -import com.webank.wedatasphere.dss.appjoint.service.SecurityService; -import com.webank.wedatasphere.dss.appjoint.visualis.service.VisualisNodeService; -import com.webank.wedatasphere.dss.appjoint.visualis.service.VisualisSecurityService; -import com.webank.wedatasphere.dss.appjoint.visualis.execution.VisualisNodeExecution; - -import java.util.Map; - -/** - * Created by enjoyyin on 2019/11/6. - */ -public class VisualisAppJoint extends AppJointUrlImpl implements AppJoint { - - private SecurityService securityService; - - private NodeExecution nodeExecution; - - private NodeService nodeService; - - @Override - public String getAppJointName() { - return "visualis"; - } - - @Override - public void init(String baseUrl, Map params) { - securityService = new VisualisSecurityService(); - securityService.setBaseUrl(baseUrl); - nodeExecution = new VisualisNodeExecution(); - nodeExecution.setBaseUrl(baseUrl); - nodeService = new VisualisNodeService(); - nodeService.setBaseUrl(baseUrl); - } - - @Override - public SecurityService getSecurityService() { - return securityService; - } - - @Override - public NodeService getNodeService() { - return nodeService; - } - - @Override - public NodeExecution getNodeExecution() { - return nodeExecution; - } -} diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisNodeService.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisNodeService.java deleted file mode 100644 index ac2c5aa8b..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisNodeService.java +++ /dev/null @@ -1,70 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.service; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.execution.core.AppJointNode; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.appjoint.service.NodeService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.appjoint.visualis.service.nodeservice.DashboardNodeService; -import com.webank.wedatasphere.dss.appjoint.visualis.service.nodeservice.DisplayNodeService; - -import java.util.Map; - -/** - * Created by enjoyyin on 2019/11/5. - */ -public class VisualisNodeService extends AppJointUrlImpl implements NodeService { - - @Override - public Map createNode(Session session, AppJointNode node, - Map requestBody) throws AppJointErrorException { - if (DisplayNodeService.getNodeType().equals(node.getNodeType())) { - return DisplayNodeService.createNode(session, getBaseUrl(), String.valueOf(node.getProjectId()), node.getNodeType(), requestBody); - } else if (DashboardNodeService.getNodeType().equals(node.getNodeType())) { - return DashboardNodeService.createNode(session, getBaseUrl(), String.valueOf(node.getProjectId()), node.getNodeType(), requestBody); - } else { - throw new AppJointErrorException(42002, "cannot recognize the nodeType " + node.getNodeType()); - } - } - - @Override - public void deleteNode(Session session, AppJointNode node) throws AppJointErrorException { - if (DisplayNodeService.getNodeType().equals(node.getNodeType())) { - DisplayNodeService.deleteNode(session, getBaseUrl(), String.valueOf(node.getProjectId()), node.getNodeType(), node.getJobContent()); - } else if (DashboardNodeService.getNodeType().equals(node.getNodeType())) { - DashboardNodeService.deleteNode(session, getBaseUrl(), String.valueOf(node.getProjectId()), node.getNodeType(), node.getJobContent()); - } else { - throw new AppJointErrorException(42002, "cannot recognize the nodeType " + node.getNodeType()); - } - } - - @Override - public Map updateNode(Session session, AppJointNode node, - Map requestBody) throws AppJointErrorException { - if (DisplayNodeService.getNodeType().equals(node.getNodeType())) { - return DisplayNodeService.updateNode(session, getBaseUrl(), node.getProjectId(), node.getNodeType(), requestBody); - } else if (DashboardNodeService.getNodeType().equals(node.getNodeType())) { - return DashboardNodeService.updateNode(session, getBaseUrl(), node.getProjectId(), node.getNodeType(), requestBody); - } else { - throw new AppJointErrorException(42002, "cannot recognize the nodeType " + node.getNodeType()); - } - } -} \ No newline at end of file diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisSecurityService.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisSecurityService.java deleted file mode 100644 index 8b970d9cb..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisSecurityService.java +++ /dev/null @@ -1,42 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.service; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.AppJointUrlImpl; -import com.webank.wedatasphere.dss.appjoint.service.SecurityService; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; - -/** - * Created by shanhuang on 2019/10/12. - */ -public class VisualisSecurityService extends AppJointUrlImpl implements SecurityService { - @Override - public Session login(String user) throws AppJointErrorException { - VisualisSession visualisSession = new VisualisSession(); - visualisSession.setUser(user); - visualisSession.getParameters().put("Token-User",user); - visualisSession.getParameters().put("Token-Code","WS-AUTH"); - return visualisSession; - } - - @Override - public void logout(String user) { - - } -} diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisSession.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisSession.java deleted file mode 100644 index ae06d1c8f..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/VisualisSession.java +++ /dev/null @@ -1,74 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.service; - -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import org.apache.http.Header; -import org.apache.http.cookie.Cookie; - -import java.util.HashMap; -import java.util.Map; - -/** - * Created by shanhuang on 2019/10/12. - */ -public class VisualisSession implements Session { - private long lastAccessTime; - private Cookie[] cookies; - private Header[] headers; - private String user; - private Map parameters = new HashMap<>(); - - @Override - public String getUser() { - return user; - } - - public void setUser(String user) { - this.user = user; - } - - @Override - public Cookie[] getCookies() { - return this.cookies; - } - - @Override - public Header[] getHeaders() { - return this.headers; - } - - @Override - public Map getParameters() { - return this.parameters; - } - - @Override - public long getLastAccessTime() { - return this.lastAccessTime; - } - - public void setCookies(Cookie[] cookies){ - this.cookies = cookies; - } - - @Override - public void updateLastAccessTime() { - lastAccessTime = System.currentTimeMillis(); - } -} diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/DashboardNodeService.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/DashboardNodeService.java deleted file mode 100644 index 9f2b42efb..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/DashboardNodeService.java +++ /dev/null @@ -1,151 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.service.nodeservice; - -import com.fasterxml.jackson.core.JsonProcessingException; -import com.google.gson.Gson; -import com.google.gson.JsonElement; -import com.google.gson.JsonObject; -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; -import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.IOException; -import java.util.HashMap; -import java.util.Map; -/** - * Created by shanhuang on 2019/10/12. - */ -public class DashboardNodeService { - private final static String dashboardUrl = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/dashboardPortals"; - private final static Logger logger = LoggerFactory.getLogger(DashboardNodeService.class); - - public static Map createNode(Session session, String url, String projectId, String nodeType, - Map requestBody) throws AppJointErrorException{ - Map element; - try { - HttpUtils httpUtils = new HttpUtils(); - requestBody.put("projectId", Integer.valueOf(projectId)); - String jsonParams = BDPJettyServerHelper.jacksonJson().writeValueAsString(requestBody); - String resultString = httpUtils.sendPostReq(session, url + dashboardUrl, jsonParams, session.getUser()); - Map jsonObject = BDPJettyServerHelper.jacksonJson().readValue(resultString, Map.class); - Map header = (Map) jsonObject.get("header"); - int code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - int dashboardId = (int) ((Map) jsonObject.get("payload")).get("id"); - Map reqMap = new HashMap<>(); - reqMap.put("config", ""); - reqMap.put("dashboardPortalId", dashboardId); - reqMap.put("index", 0); - reqMap.put("name", requestBody.get("name").toString()); - reqMap.put("parentId", 0); - reqMap.put("type", 1); - String reqJsonObject = BDPJettyServerHelper.jacksonJson().writeValueAsString(reqMap); - String result = httpUtils.sendPostReq(session, url + dashboardUrl + "/" + dashboardId + "/dashboards", reqJsonObject, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(result, Map.class); - Map header1 = (Map) element.get("header"); - code = (int) header1.get("code"); - if (code != 200) { - String errorMsg = header1.toString(); - throw new AppJointErrorException(code, errorMsg); - } - element.put("dashboardPortalId", dashboardId); - }catch (Exception ex){ - throw new AppJointErrorException(90155,"Update Display AppJointNode Exception",ex); - } - - return element; - } - - public static Map deleteNode(Session session, String url, String projectId, String nodeType, - Map requestBody) throws AppJointErrorException{ - Map element; - try { - HttpUtils httpUtils = new HttpUtils(); - requestBody.put("projectId", Integer.valueOf(projectId)); - logger.info("DashboardNodeServiceImpl request params is " + requestBody + ",nodeType:" + nodeType); - logger.info("Dashboard url is " + url); - String nodeId = requestBody.get("id").toString(); - String resultString = httpUtils.sendHttpDelete(session, url + dashboardUrl + "/" + nodeId, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(resultString, Map.class); - Map header = (Map) element.get("header"); - int code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - }catch (Exception ex){ - throw new AppJointErrorException(90156,"Update Display AppJointNode Exception",ex); - } - return element; - } - - public static Map updateNode(Session session, String url, Long projectId, String nodeType, - Map requestBody) throws AppJointErrorException{ - Map element; - try { - HttpUtils httpUtils = new HttpUtils(); - requestBody.put("projectId", projectId.intValue()); - String jsonParams = BDPJettyServerHelper.jacksonJson().writeValueAsString(requestBody); - logger.info("updateNode request params is " + jsonParams + ",nodeType:" + nodeType); - logger.info("Dashboard url is " + url); - String dashboardPortalId = requestBody.get("dashboardPortalId").toString(); - String resultString = httpUtils.sendHttpPut(session, url + dashboardUrl + "/" + dashboardPortalId + "/dashboards", jsonParams, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(resultString, Map.class); - Map header = (Map) element.get("header"); - int code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } -// int dashboardId = jsonObject.get("payload").getAsJsonObject().get("id").getAsInt(); - Map reqMap = new HashMap<>(); - reqMap.put("avatar", "9"); - reqMap.put("description", ""); - reqMap.put("id", dashboardPortalId); - reqMap.put("name", requestBody.get("name").toString()); - reqMap.put("publish", true); - reqMap.put("roleIds", "[]"); - String reqJson = BDPJettyServerHelper.jacksonJson().writeValueAsString(reqMap); - logger.info("updateNode request slides params is: " + reqJson); - String result = httpUtils.sendHttpPut(session, url + dashboardUrl + "/" + dashboardPortalId, reqJson, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(result, Map.class); - header = (Map) element.get("header"); - code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - element.put("dashboardPortalId", dashboardPortalId); - }catch (Exception ex){ - throw new AppJointErrorException(90157,"Update Display AppJointNode Exception",ex); - } - - return element; - } - - public static String getNodeType() { - return "linkis.appjoint.visualis.dashboard"; - } -} diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/DisplayNodeService.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/DisplayNodeService.java deleted file mode 100644 index 1f21dd3f6..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/DisplayNodeService.java +++ /dev/null @@ -1,125 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.service.nodeservice; - - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.linkis.server.BDPJettyServerHelper; -import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import java.util.HashMap; -import java.util.Map; -/** - * Created by shanhuang on 2019/10/12. - */ -public class DisplayNodeService { - private final static String displayUrl = "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION() + "/visualis/displays"; - private final static String config = "{\"slideParams\":{\"width\":1920,\"height\":1080,\"backgroundColor\":[255,255,255],\"scaleMode\":\"noScale\",\"backgroundImage\":null}}"; - private final static Logger logger = LoggerFactory.getLogger(DisplayNodeService.class); - - public static Map createNode(Session session, String url, String projectId, - String nodeType, Map requestBody) throws AppJointErrorException { - Map element; - try { - requestBody.put("projectId", Integer.valueOf(projectId)); - String jsonParams = BDPJettyServerHelper.jacksonJson().writeValueAsString(requestBody); - logger.info("DisplayNodeServiceImpl request params is " + jsonParams + ",nodeType:" + nodeType); - String displayRes = HttpUtils.sendPostReq(session, url + displayUrl, jsonParams, session.getUser()); - logger.info("Create Display AppJointNode First Return Value: " + displayRes); - Map jsonObject = BDPJettyServerHelper.jacksonJson().readValue(displayRes, Map.class); - Map header = (Map) jsonObject.get("header"); - int code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - int displayId = (int) ((Map) jsonObject.get("payload")).get("id"); - Map reqMap = new HashMap<>(); - reqMap.put("config", config); - reqMap.put("displayId", displayId); - reqMap.put("index", 0); - String reqJsonObject = BDPJettyServerHelper.jacksonJson().writeValueAsString(reqMap); - logger.info("request slides params is: " + reqJsonObject); - String result = HttpUtils.sendPostReq(session, url + displayUrl + "/" + displayId + "/slides", reqJsonObject, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(result, Map.class); - header = (Map) element.get("header"); - code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - - }catch (Exception ex){ - throw new AppJointErrorException(90173,"Create Display AppJointNode Exception",ex); - } - - return element; - - } - - - public static Map deleteNode(Session session, String url, String projectId, - String nodeType, Map requestBody) throws AppJointErrorException{ - Map element; - try { - requestBody.put("projectId", Integer.valueOf(projectId)); - logger.info("request params is " + requestBody + ",nodeType:" + nodeType); - logger.info("Delete display url is " + url); - String nodeId = requestBody.get("id").toString(); - String resultString = HttpUtils.sendHttpDelete(session, url + displayUrl + "/" + nodeId, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(resultString, Map.class); - Map header = (Map) element.get("header"); - int code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - }catch (Exception ex){ - throw new AppJointErrorException(90174,"Delete Display AppJointNode Exception",ex); - } - return element; - } - - - public static Map updateNode(Session session, String url, Long projectId, - String nodeType, Map requestBody) throws AppJointErrorException{ - Map element; - try { - logger.info("[updateNode]: request projectId is {}, nodeType is {}, params: {}.", projectId, nodeType, requestBody); - String nodeId = requestBody.get("id").toString(); - String jsonParams = BDPJettyServerHelper.jacksonJson().writeValueAsString(requestBody); - String displayRes = HttpUtils.sendHttpPut(session, url + displayUrl + "/" + nodeId, jsonParams, session.getUser()); - element = BDPJettyServerHelper.jacksonJson().readValue(displayRes, Map.class); - Map header = (Map) element.get("header"); - int code = (int) header.get("code"); - if (code != 200) { - String errorMsg = header.toString(); - throw new AppJointErrorException(code, errorMsg); - } - }catch (Exception ex){ - throw new AppJointErrorException(90175,"Update Display AppJointNode Exception",ex); - } - return element; - } - - public static String getNodeType() { - return "linkis.appjoint.visualis.display"; - } -} diff --git a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/HttpUtils.java b/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/HttpUtils.java deleted file mode 100644 index 4a030dc68..000000000 --- a/visualis-appjoint/appjoint/src/main/java/com/webank/wedatasphere/dss/appjoint/visualis/service/nodeservice/HttpUtils.java +++ /dev/null @@ -1,153 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.service.nodeservice; - -import com.webank.wedatasphere.dss.appjoint.exception.AppJointErrorException; -import com.webank.wedatasphere.dss.appjoint.service.session.Session; -import com.webank.wedatasphere.dss.appjoint.visualis.service.VisualisSession; -import org.apache.commons.io.IOUtils; -import org.apache.http.HttpEntity; -import org.apache.http.client.CookieStore; -import org.apache.http.client.methods.CloseableHttpResponse; -import org.apache.http.client.methods.HttpDelete; -import org.apache.http.client.methods.HttpPost; -import org.apache.http.client.methods.HttpPut; -import org.apache.http.entity.StringEntity; -import org.apache.http.impl.client.BasicCookieStore; -import org.apache.http.impl.client.CloseableHttpClient; -import org.apache.http.impl.client.HttpClients; -import org.apache.http.protocol.HTTP; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.UnsupportedEncodingException; -/** - * Created by shanhuang on 2019/10/12. - */ -public class HttpUtils { - private static Logger logger = LoggerFactory.getLogger(HttpUtils.class); - public static String sendPostReq(Session session, String url, String params, - String user) throws AppJointErrorException { - String resultString = "{}"; - VisualisSession visualisSession = (VisualisSession)session; - String token = visualisSession.getParameters().get("Token-Code"); - logger.info("sendPostReq url is: "+url+",session:"+token ); - HttpPost httpPost = new HttpPost(url); - httpPost.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - httpPost.addHeader("Token-User",user); - httpPost.addHeader("Token-Code",token); - CookieStore cookieStore = new BasicCookieStore(); -// cookieStore.addCookie(session.getCookies()[0]); - logger.info("Http request params is :"+params); - StringEntity entity = null; - try { - entity = new StringEntity(params); - } catch (UnsupportedEncodingException e) { - throw new AppJointErrorException(42000, "create StringEntity failed!", e); - } - entity.setContentEncoding("UTF-8"); - entity.setContentType("application/json");//发送json数据需要设置contentType - httpPost.setEntity(entity); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpPost); - HttpEntity ent = response.getEntity(); - resultString = IOUtils.toString(ent.getContent(), "utf-8"); - logger.info("Send Http Request Success", resultString); - } catch (Exception e) { - logger.error("Send Http Request Failed", e); - throw new AppJointErrorException(42000, e.getMessage(), e); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - return resultString; - } - - public static String sendHttpDelete(Session session,String url,String user) throws AppJointErrorException { - String resultString = "{}"; - HttpDelete httpdelete = new HttpDelete(url); - //设置header - VisualisSession visualisSession = (VisualisSession)session; - String token = visualisSession.getParameters().get("Token-Code"); - logger.info("sendDeleteReq url is: "+url+",session:"+token ); - httpdelete.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - httpdelete.addHeader("Token-User",user); - httpdelete.addHeader("Token-Code",token); - CookieStore cookieStore = new BasicCookieStore(); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpdelete); - HttpEntity ent = response.getEntity(); - resultString = IOUtils.toString(ent.getContent(), "utf-8"); - logger.info("Send Http Delete Request Success", resultString); - } catch (Exception e) { - logger.error("Send Http Delete Request Failed", e); - throw new AppJointErrorException(42001, e.getMessage(), e); - }finally{ - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - return resultString; - } - - public static String sendHttpPut(Session session, String url, String params, - String user) throws AppJointErrorException { - String resultString = "{}"; - VisualisSession visualisSession = (VisualisSession)session; - String token = visualisSession.getParameters().get("Token-Code"); - logger.info("sendPostReq url is: "+url+",session:"+token ); - HttpPut httpPut = new HttpPut(url); - httpPut.addHeader(HTTP.CONTENT_ENCODING, "UTF-8"); - httpPut.addHeader("Token-User",user); - httpPut.addHeader("Token-Code",token); - CookieStore cookieStore = new BasicCookieStore(); -// cookieStore.addCookie(session.getCookies()[0]); - logger.info("Http put params is :"+params); - StringEntity entity = null; - try { - entity = new StringEntity(params); - } catch (UnsupportedEncodingException e) { - throw new AppJointErrorException(42000, "create StringEntity failed!", e); - } - entity.setContentEncoding("UTF-8"); - entity.setContentType("application/json");//发送json数据需要设置contentType - httpPut.setEntity(entity); - CloseableHttpClient httpClient = null; - CloseableHttpResponse response = null; - try { - httpClient = HttpClients.custom().setDefaultCookieStore(cookieStore).build(); - response = httpClient.execute(httpPut); - HttpEntity ent = response.getEntity(); - resultString = IOUtils.toString(ent.getContent(), "utf-8"); - logger.info("Send Http Put Success", resultString); - } catch (Exception e) { - logger.error("Send Http Put Failed", e); - throw new AppJointErrorException(42000, e.getMessage(), e); - } finally { - IOUtils.closeQuietly(response); - IOUtils.closeQuietly(httpClient); - } - return resultString; - } - -} diff --git a/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/visualis/execution/VisualisNodeExecution.scala b/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/visualis/execution/VisualisNodeExecution.scala deleted file mode 100644 index fa2575726..000000000 --- a/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/visualis/execution/VisualisNodeExecution.scala +++ /dev/null @@ -1,142 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.execution - -import java.io.{ByteArrayOutputStream, InputStream} -import java.util -import java.util.Base64 - -import com.webank.wedatasphere.dss.appjoint.execution.NodeExecution -import com.webank.wedatasphere.dss.appjoint.execution.common.{CompletedNodeExecutionResponse, NodeExecutionResponse} -import com.webank.wedatasphere.dss.appjoint.execution.core.{AppJointNode, CommonAppJointNode, NodeContext} -import com.webank.wedatasphere.dss.appjoint.service.session.Session -import com.webank.wedatasphere.dss.appjoint.visualis.execution.VisualisNodeExecutionConfiguration._ -import com.webank.wedatasphere.linkis.common.exception.ErrorException -import com.webank.wedatasphere.linkis.common.log.LogUtils -import com.webank.wedatasphere.linkis.common.utils.{Logging, Utils} -import com.webank.wedatasphere.linkis.storage.{LineMetaData, LineRecord} -import org.apache.commons.io.IOUtils - -import scala.collection.JavaConversions.mapAsScalaMap -import scala.concurrent.duration.Duration -import scala.concurrent.{Await, ExecutionContext} -import dispatch._ -import org.json4s.{DefaultFormats, Formats} - -/** - * Created by enjoyyin on 2019/10/12. - */ -class VisualisNodeExecution extends NodeExecution with Logging { - - private val DISPLAY = "display" - private val DASHBOARD = "dashboard" - - var basicUrl:String = _ - - protected implicit val executors: ExecutionContext = Utils.newCachedExecutionContext(VISUALIS_THREAD_MAX.getValue, getName + "-NodeExecution-Thread", true) - protected implicit val formats: Formats = DefaultFormats - - private implicit def svc(url: String): Req = - dispatch.url(url) - - - - override def getBaseUrl: String = this.basicUrl - - override def setBaseUrl(basicUrl: String): Unit = this.basicUrl = basicUrl - - def getName:String = "visualis" - - override def canExecute(node: AppJointNode, nodeContext: NodeContext, session: Session): Boolean = node.getNodeType.toLowerCase.contains(getName.toLowerCase) - - override def execute(node: AppJointNode, nodeContext: NodeContext, session: Session): NodeExecutionResponse = node match { - case commonAppJointNode: CommonAppJointNode => - val appJointResponse = new CompletedNodeExecutionResponse() - val idMap = commonAppJointNode.getJobContent - val id = idMap.values().iterator().next().toString - val url = if(commonAppJointNode.getNodeType.toLowerCase.contains(DISPLAY)) getDisplayPreviewUrl(nodeContext.getGatewayUrl, id) - else if(commonAppJointNode.getNodeType.toLowerCase.contains(DASHBOARD)) getDashboardPreviewUrl(nodeContext.getGatewayUrl, id) - else { - appJointResponse.setIsSucceed(false) - appJointResponse.setErrorMsg("不支持的appJoint类型：" + node.getNodeType) - return appJointResponse - } - var response = "" - val headers = nodeContext.getTokenHeader(nodeContext.getUser) - nodeContext.appendLog(LogUtils.generateInfo(s"Ready to download preview picture from $url.")) - Utils.tryCatch(download(url, null, headers.toMap, - input => Utils.tryFinally{ - val os = new ByteArrayOutputStream() - IOUtils.copy(input, os) - response = new String(Base64.getEncoder.encode(os.toByteArray)) - //response = IOUtils.toString(input, ServerConfiguration.BDP_SERVER_ENCODING.getValue) - }(IOUtils.closeQuietly(input)))){ t => - val errException = new ErrorException(70063, "failed to do visualis request") - errException.initCause(t) - appJointResponse.setException(errException) - appJointResponse.setIsSucceed(false) - appJointResponse.setErrorMsg(s"用户${nodeContext.getUser}请求Visualis失败！URL为: " + url) - return appJointResponse - } - nodeContext.appendLog(LogUtils.generateInfo("Preview picture downloaded, now ready to write results.")) - val imagesBytes = response - val resultSetWriter = nodeContext.createPictureResultSetWriter() - Utils.tryFinally{ - resultSetWriter.addMetaData(new LineMetaData()) - resultSetWriter.addRecord(new LineRecord(imagesBytes)) - }(IOUtils.closeQuietly(resultSetWriter)) - appJointResponse.setIsSucceed(true) - appJointResponse - } - - def download(url: String, queryParams: Map[String, String], headerParams: Map[String, String], - write: InputStream => Unit, - paths: String*): Unit = { - var req = url.GET - if(headerParams != null && headerParams.nonEmpty) req = req <:< headerParams - if(queryParams != null) queryParams.foreach{ case (k, v) => req = req.addQueryParameter(k, v)} - if(paths != null) paths.filter(_ != null).foreach(p => req = req / p) - val response = Http(req OK as.Response(_.getResponseBodyAsStream)).map(write) - Await.result(response, Duration.Inf) - } - - private def getRealId(displayId:String):Int = { - Utils.tryCatch{ - val f = java.lang.Float.parseFloat(displayId) - f.asInstanceOf[Int] - }{ - _ => 0 - } - } - - def getPreviewUrl(gatewayUrl: String, uri: String, displayId: String): String = { - val realId = getRealId(displayId) - val realGatewayUrl = "http://" + gatewayUrl - if(!realGatewayUrl.endsWith("/") && !uri.startsWith("/")) - realGatewayUrl + "/" + String.format(uri, realId.toString) - else if(realGatewayUrl.endsWith("/") && uri.startsWith("/")) - realGatewayUrl.substring(0, realGatewayUrl.length - 1) + String.format(uri, realId.toString) - else realGatewayUrl + String.format(uri, realId.toString) - } - - def getDisplayPreviewUrl(gatewayUrl: String, displayId: String): String = getPreviewUrl(gatewayUrl, DISPLAY_PREVIEW_URL_FORMAT.getValue, displayId) - - def getDashboardPreviewUrl(gatewayUrl: String, dashboardId: String): String = getPreviewUrl(gatewayUrl, DASHBOARD_PREVIEW_URL_FORMAT.getValue, dashboardId) - - override def init(map: util.Map[String, AnyRef]): Unit = {} -} diff --git a/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/visualis/execution/VisualisNodeExecutionConfiguration.scala b/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/visualis/execution/VisualisNodeExecutionConfiguration.scala deleted file mode 100644 index 95c3648ea..000000000 --- a/visualis-appjoint/appjoint/src/main/scala/com/webank/wedatasphere/dss/appjoint/visualis/execution/VisualisNodeExecutionConfiguration.scala +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Copyright 2019 WeBank - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - * - */ - -package com.webank.wedatasphere.dss.appjoint.visualis.execution - -import com.webank.wedatasphere.linkis.common.conf.CommonVars -import com.webank.wedatasphere.linkis.server.conf.ServerConfiguration - -/** - * Created by enjoyyin on 2019/10/12. - */ -object VisualisNodeExecutionConfiguration { - - val DISPLAY_PREVIEW_URL_FORMAT = CommonVars("wds.dss.appjoint.visualis.display.preview.url", - "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION + "/visualis/displays/%s/preview") - val DASHBOARD_PREVIEW_URL_FORMAT = CommonVars("wds.dss.appjoint.visualis.dashboard.preview.url", - "/api/rest_s/" + ServerConfiguration.BDP_SERVER_VERSION + "/visualis/dashboard/%s/preview") - - val VISUALIS_THREAD_MAX = CommonVars("wds.dss.appjoint.visualis.thread.max", 20) - -} diff --git a/web/LICENSE b/web/LICENSE new file mode 100755 index 000000000..261eeb9e9 --- /dev/null +++ b/web/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/web/README-DEV.md b/web/README-DEV.md new file mode 100755 index 000000000..8a176a51d --- /dev/null +++ b/web/README-DEV.md @@ -0,0 +1,89 @@ +### 1.0.0版本重构说明 + +由于DSS里存在相对独立的一些功能模块（如scriptis、workflow、管理控制台等），之前是一个整体大的单页应用开发模式，虽然在具体业务逻辑上也是模块化开发，但这些模块都混在一起，在迭代开发过程中暴露一些问题。比如社区需要贡献代码，对这些修改的代码影响范围比较难以评估，同时存在跨模块的组件引用，对于新增需求开发时需要对整体应用比较熟悉，对于刚加入的开发同事要熟悉整个应用也有很大挑战。同时DSS有接入第三方产品系统的需求，这个比较符合微前端的应用场景。所以有必要拆分应用。 + +### 重构目的 + +- 前端工程管理方面达到目录结构清晰，方便维护及迭代开发 +- 后续微前端的实施 +- 插件开发支持 + +### 重构目标 + +鉴于目前DSS现状及后续规划要求，决定采用 一个主应用+多个子应用 的方式来重构前端工程 +主应用部分负责提供应用入口，组织整个应用。子应用必须是可以看作是一个单独的应用，并且各子应用之间应当明确边界， +在设计开发层面应没有相互耦合依赖关系。 +主应用需要提供应用入口，通过一个配置文件管理子应用的加载。各子应用提供路由、国际化等配置在主应用启动时会被合并 +提供可扩展的插件化开发支持 + +### 项目结构 + +``` +├─dist # 构建后静态资源 +├─docs # 文档 +├─node_modules +├─patches # 修复第三方依赖 +├─public # 公共index.html +├─scripts # 工具脚本 +└─src + ├─apps # dss各应用存放目录，每个应用独立 + │ ├─apiServices # 数据服务相关功能 + │ ├─scriptis # scriptis相关功能 + │ │ ├─assets # 应用所需的图片、css等资源 + │ │ ├─config # + │ │ ├─i18n # 国际化的中英文json + │ │ ├─module # 当前应用所需的模块，每个模块相对独立，模块私有资源内置 + │ │ ├─service + │ │ └─view # 当前应用的页面，路由在同级目录下router.js下配置 + │ ├─workspace # 工作空间应用 + │ │ ├─assets # 静态资源 + │ │ ├─i18n # 当前应用所需的国际化翻译json + │ │ ├─module # 当前应用所需的模块 + │ │ └─view + │ └─workflows + ├─common # 全局使用的公共服务和方法 + ├─components # 全局使用的公共组件部分 + ├─dss # 项目的顶级根应用，apps里的应用都是其子路由 + ├─config.json # 子应用配置 + ├─dynamic-apps.js # 动态应用模块合并 + ├─main.js # 主应用启动入口 + └─router.js # 合并后的应用路由 +``` + +### 建议/约束 + +新增功能模块先确定涉及应用，按照上面目录结构维护代码同时建议遵守以下约束： + +- 子应用可以配置自己的layout需要在应用router模块导出配置subAppRoutes，参考scriptis +- 子应用支持使用自己的header，需要在config.json里配置模块路径 +- 各应用需要使用iView作为UI库，并提供路由，国际化等配置写入config.json +- 各应用间不可以相互依赖 +- 可复用组件，资源需要合理放置 +- 各应用路由应以应用名做统一前缀 +- 各应用之间需要事件通信，应当在config.json 里声明对应module文件路径 +- 新增功能模块需要按照现有目录约束建立文件，已有功能修改应在有限模块内进行，控制影响范围 +- 全局共用组件、公共基础样式、工具方法修改需评估后才能修改，并且重点review + +### 待优化项 + +1. src/components 公共组件需要review，这部分公共组件应对大量数据的情况做性能方面的优化 + +### 如何新增一个子应用 + +1. config.json 新增应用配置 +2. src/apps 下新建应用目录进行应用开发 + + +### 前端开发、构建打包 + +``` +# 开发启动DSS +npm run serve +# 子应用开发 +npm run serve --module=scriptis,workflows --micro_module=workflows +# 打包DSS应用 +npm run build +# 打包子应用，支持通过module组合 +npm run build --module=scriptis,workflows --micro_module=workflows +npm run build --module=apiServices,workspace --micro_module=apiServices +``` \ No newline at end of file diff --git a/web/README.md b/web/README.md new file mode 100755 index 000000000..f6b4f90e8 --- /dev/null +++ b/web/README.md @@ -0,0 +1,45 @@ +Scriptis +============ + +[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) + +English | [Chinese](docs/zh_CN/README.md) + +## Introduction + +Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis. + +![running](docs/en_US/images/readme/running.gif) + +## Features + +* Script editor: Support multi-language, auto-completion, syntax highlighting and SQL syntax error-correction. + +* Computation engines: Based on Linkis, Scriptis connects with multiple computation engines such as Spark, Hive, Python, etc. + +* Runtime functionality: Complete job life cycle display and intelligent diagnosis. + +* Result set: Multiple result sets support, customized result set alias and one-click visualization. + +* Database Services: Functionalities for database management and files(CVS, Excel) import/export to/from tables. + +* Context: UDFs, custom variables and functions management and sharing. + +* Console: Customized settings for engine parameters, task/engine management and resource isolation/display. + + +## QuickStart + +Read the Quick Start [Quick Start](/docs/en_US/ch3/Scriptis_Quick_Start.md) + +## Comparison with similar scheduler systems +![Comparison](/docs/en_US/images/readme/Comparison.png) + +## Community +If you desire immediate response, please kindly raise issues to us or scan the below QR code by WeChat and QQ to join our group: + +![WeChatQQ](/docs/en_US/images/wechatQQ.png) + +## License + +Scriptis is under the Apache 2.0 license. See the [LICENSE]((http://www.apache.org/licenses/LICENSE-2.0)) file for details diff --git a/web/babel.config.js b/web/babel.config.js old mode 100644 new mode 100755 diff --git a/web/cn.json b/web/cn.json deleted file mode 100644 index 74b7a496c..000000000 --- a/web/cn.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "dirPath": "src\\js\\module\\resourceSimple", - "resourceSimple": { - "YS": "意书", - "FL": "分类", - "ZH": "知画", - "ZWSJ": "暂无数据", - "KX": "空闲", - "FM": "繁忙", - "QD": "启动", - "LX": "按类别", - "AZT": "按状态", - "DDJK": "请等待接口返回！", - "QDYQWFJS": "启动状态引擎无法结束！", - "WXZYQ": "未选中引擎！", - "RWGLQ": "任务管理器", - "YQGLQ": "引擎管理器", - "DLGLQ": "队列管理器", - "SX": "刷新", - "JSRW": "结束任务", - "JSYQ": "结束引擎", - "PDZ": "排队中", - "ZYSQ": "资源申请中", - "YX": "运行", - "YQ": "引擎", - "RW": "任务", - "WZJBMC": "未知脚本名称", - "ZHJB": "知画脚本", - "YQRW": "引擎和任务", - "JSYQRW": "结束引擎和任务", - "CKJBXX": "查看脚本信息", - "FZJBXX": "复制脚本信息", - "QXZYTJL": "请选择一条记录！", - "WHQD": "未获取到未获取到strongerExecId，请联系管理员查看！", - "WHQDENGINE": "未获取到engineInstance，请联系管理员查看！", - "YXTS": "运行提示", - "YJTZZXJB": "已经停止执行脚本", - "JSYQHRWCG": "结束引擎和任务成功", - "DKCG": "打开成功", - "JBFZDZTRB": "脚本信息已复制到粘贴板，请使用ctrl+v组合键粘贴！", - "XZDL": "请选择队列", - "ZYSYL": "资源使用率", - "H": "核", - "NC": "内存", - "ZYSYPHB": "资源使用排行榜", - "ZW": "暂无" - } -} diff --git a/web/config.sh b/web/config.sh old mode 100644 new mode 100755 index 864e49271..0ebe48dbe --- a/web/config.sh +++ b/web/config.sh @@ -1,8 +1,8 @@ -#dss web port -dss_web_port="8088" +#Configuring front-end ports +dss_port="8088" -#dss web access linkis gateway adress -linkis_gateway_url="http://localhost:9001" +#URL of the backend linkis gateway +linkis_url="http://localhost:20401" -#dss nginx ip -dss_nginx_ip=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}'|awk 'NR==1') +#dss ip address +dss_ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}') diff --git a/web/docs/en_US/README.md b/web/docs/en_US/README.md new file mode 100755 index 000000000..6ab30f1ad --- /dev/null +++ b/web/docs/en_US/README.md @@ -0,0 +1,34 @@ +## Introduction + +Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis. + +## Features + +* Script editor: Support multi-language, auto-completion, syntax highlighting and SQL syntax error-correction. + +* Computation engines: Based on Linkis, Scriptis connects with multiple computation engines such as Spark, Hive, Python, etc. + +* Runtime functionality: Complete job life cycle display and intelligent diagnosis. + +* Result set: Multiple result sets support, customized result set alias and one-click visualization. + +* Database Services: Functionalities for database management and files(CVS, Excel) import/export to/from tables. + +* Context: UDFs, custom variables and functions management and sharing. + +* Console: Customized settings for engine parameters, task/engine management and resource isolation/display. + + +## Comparison with similar scheduler systems +![Comparison](/docs/en_US/images/readme/Comparison.png) + +## Document +* [Front-end_deployment_documentation](ch1/Front-end_deployment_documentation.md) +* [Compilation](ch2/Compilation.md) +* [Scriptis_Quick_Start](ch3/Scriptis_Quick_Start.md) +* [Scriptis_Manual](ch4/Scriptis_Manual.md) + + +## License + +Scriptis is under the Apache 2.0 license. See the [LICENSE]((http://www.apache.org/licenses/LICENSE-2.0)) file for details \ No newline at end of file diff --git a/web/docs/en_US/SUMMARY.md b/web/docs/en_US/SUMMARY.md new file mode 100755 index 000000000..2b829a4a8 --- /dev/null +++ b/web/docs/en_US/SUMMARY.md @@ -0,0 +1,10 @@ +# Summary +* [Introduction](README.md) +* [ch1 Front-end_deployment_documentation]() + * [Front-end_deployment_documentation](ch1/Front-end_deployment_documentation.md) +* [ch2 Compilation ]() + * [Compilation](ch2/Compilation.md) +* [ch3 Scriptis_Quick_Start]() + * [Scriptis_Quick_Start](ch3/Scriptis_Quick_Start.md) +* [ch4 Scriptis_Manual ]() + * [Scriptis_Manual](ch4/Scriptis_Manual.md) diff --git a/web/docs/en_US/ch1/Front-end_deployment_documentation.md b/web/docs/en_US/ch1/Front-end_deployment_documentation.md new file mode 100755 index 000000000..5fa0ac685 --- /dev/null +++ b/web/docs/en_US/ch1/Front-end_deployment_documentation.md @@ -0,0 +1,114 @@ +Scriptis is a data analysis tool based on Linkis. Before deploying Scriptis, you need to deploy Linkis first. For the Linkis deploy document, see: [Linkis DeployDoc](https://github.com/WeBankFinTech/Linkis/blob/master/docs/en_US/ch1/deploy.md) + +## 1 Preparation + +1. Select the corresponding installation package to download. + +2. Unzip the downloaded installation package in the installation directory: unzip wedatasphere-scriptis-0.7.0-dist.zip. + +## 2 Deploy + +​ There are two deployment methods, automated and manual deployment. + +### 2.1 Automated deployment + +Go to the frontend directory ```wedatasphere-scriptis``` and edit ```vi config.sh ``` to change the interface address of the frontend and backend port. backend port interface address is the gateway address of linkis. + +### (3) Modify and save the configuration file created above + +``` +# Configuring front-end ports +scriptis_port="8088" + +# URL of the backend linkis gateway +linkis_url="http://localhost:20401" + +# Scriptis ip address +scriptis_ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}') +``` + +After the modification, run the following command in the directory: ```sudo sh install.sh > install.log 2>&1``` + +Next, you can access ```http://scriptis_ipaddr:scriptis_port``` directly via Chrome, scriptis_port is the port configured in config.sh and scriptis_ipaddr is the IP of the machine that used for installation. + +If encounter access failure, please check install.log and find out the errors. + +### 2.2 Manual deployment + +1. Install Nginx: ```sudo yum install nginx -y``` + +2. Modify the configuration file:```sudo vi /etc/nginx/conf.d/scriptis.conf``` + + Add the following: + +``` +server { + listen 8080;# Access Port + server_name localhost; + #charset koi8-r; + #access_log /var/log/nginx/host.access.log main; + location / { + root /appcom/Install/scriptis/ROOT; # directory where package decompressed + #in the fronted + index index.html index.html; + } + location /ws {#webSocket configure spport + proxy_pass http://192.168.xxx.xxx:9001;#IP port of the linkis gateway service + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + location /api { + proxy_pass http://192.168.xxx.xxx:9001;#IP port of the linkis gateway service + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header x_real_ipP $remote_addr; + proxy_set_header remote_addr $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_connect_timeout 4s; + proxy_read_timeout 600s; + proxy_send_timeout 12s; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection upgrade; + } + #error_page 404 /404.html; + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + } +``` + +3. Copy the frontend package to the corresponding directory: ```/appcom/Install/scriptis/ROOT; # directory where package decompressed in the frontend``` +4. Start service: ```sudo systemctl restart nginx``` +5. You can directly access ```http://nginx_ip:nginx_port``` via Chrome after execution. + +## 3 FAQs + +(1) limitations on the size of files that being uploaded + +``` +sudo vi /etc/nginx/nginx.conf +``` + +Change the uploading size: + +``` +client_max_body_size 200m +``` + +(2) Interface timeout + +``` +sudo vi /etc/nginx/conf.d/scriptis.conf +``` + +Change the interface timeout: + +``` +proxy_read_timeout 600s +``` + diff --git a/web/docs/en_US/ch2/Compilation.md b/web/docs/en_US/ch2/Compilation.md new file mode 100755 index 000000000..1f22f0a29 --- /dev/null +++ b/web/docs/en_US/ch2/Compilation.md @@ -0,0 +1,102 @@ +# Compilation + +## Getting Started + +### Prerequisites + +Install Node.js on your computer. Download Link: [http://nodejs.cn/download/](http://nodejs.cn/download/). Recommend using the latest stable version. + +**Only do this step at the first time.** + +### Installation + +Run the following commands in terminal: + +``` +git clone https://github.com/WeBankFinTech/Scriptis.git +cd DataSphereStudio/web +npm install +``` + + Commands explanation: + +1. Pull remote repository to local: `git clone https://github.com/WeBankFinTech/Scriptis.git` + +2. Change to the root directory of the project: `cd DataSphereStudio/web` + +3. Install all dependencies required for the project: `npm install` + +**Only do this step at the first time.** + +### Configuration + +You need to make some configurations in your code, such as port address of backend server and socket address of backend server in .env.development file in root directory. + +``` +// Port address of backend server +VUE_APP_MN_CONFIG_PREFIX=http://yourIp:yourPort/yourPath +// Socket address +VUE_APP_MN_CONFIG_SOCKET=/yourSocketPath +``` + +You can refer to the official documentation of vue-cli for detailed explanation. [Modes and environment variables](https://cli.vuejs.org/guide/mode-and-env.html#modes) + +### Building project + +You can run the following command in terminal to build the project: + +``` +npm run build +``` + +A folder named "dist" would appear in your project's root directory if the command has run successfully and you can directly put "dist" to your static server. + +### How to run + +You would need to run the following command in terminal if you want to run project on your local browser and see corresponding effects after making changes to the code. + +``` +npm run serve +``` + +Access the application in browser (Chrome recommended) via link: [http://localhost:8080/](http://localhost:8080/) . + +Changes you make to the code would dynamically reflect on the +effects shown on browser when using the method described above to run project. + +**Notes: Since frontend and backend are developed separately, when running on local browser you need to allow cross domain access in order to access the port of backend server.** + +e.g. Chrome browser: + +Configuration in Windows: + +1. Close all browser windows. + +2. Create a shortcut of chrome, right-click to choose "properties" , then go to "Shortcut" tab find "Target" and add`--args --disable-web-security --user-data-dir=C:\MyChromeDevUserData` to it . +3. Use shortcut to open the browser. + +Configuration in MacOS: + +Run the following command. (You need to replace "yourname" in the path. If it's not working, check the path of MyChromeDevUserData on your machine and copy its path to the place right after "--user-data-dir=") + +``` +open -n /Applications/Google\ Chrome.app/ --args --disable-web-security --user-data-dir=/Users/yourname/MyChromeDevUserData/ +``` + +### FAQ + +#### Failed installation when running npm install + +Try to use Taobao npm mirror: + +``` +npm install -g cnpm --registry=https://registry.npm.taobao.org +``` + +Next, run the following command instead of npm install: + +``` +cnpm install +``` + +Note that you can still use `npm run serve` and `npm run build` to run and build project. \ No newline at end of file diff --git a/web/docs/en_US/ch3/Scriptis_Quick_Start.md b/web/docs/en_US/ch3/Scriptis_Quick_Start.md new file mode 100755 index 000000000..4d1dcf207 --- /dev/null +++ b/web/docs/en_US/ch3/Scriptis_Quick_Start.md @@ -0,0 +1,35 @@ + +## Quick Login +     To be convenient to users，the default login username is consistent with the Linux user which is used for deployment. For example, if user 'hadoop' is used for deployment linkis, the user can use 'hadoop' as the username, and 'hadoop' as the password to login. Note that the password is the same with the username. +First enter the address of the front-end container: 192.168.xx.xx:8888, and then enter the username/password:hadoop/hadoop + +![01](../images/ch3/01.png) +__Hint：__ Other users must use their configured LDAP passwords to login. + +## Queue Configuration +     scriptis is backed by Spark running on yarn. To use Spark properly, users must declare an available queue in their configurations. Click Control Panel --> Settings --> Yarn Queue, and then enter the name of the yarn queue to which you have the permission. +![02](../images/ch3/02.png) + +## Create New Script +      +Finished with queue configuration, you may try to create a SQL script snippet for data query. Click WorkSpace --> Personal Directory, and then right-click to create a new SQL script. +1. Create +![03](../images/ch3/03.png) +2. Create SQL script +![04](../images/ch3/04.png) +3. Type SQL code snippet +![05](../images/ch3/05.png) + + +## Click & Run +      +Click the 'Run' button above the script area to execute the script. The status of the execution can be monitored by logs. +![01](../images/ch3/06.png) + +## Result Display +      +For the first-time execution, please wait for a slightly long time after clicking the 'Run' button, as the background service is firing a new Engine for you. Result sets will be displayed after successful executions. Users can download or export the result sets. +![01](../images/ch3/07.png) + +For more detailed manual please see：[Scriptis Manual](../ch4/Scriptis_Manual.md) + diff --git a/web/docs/en_US/ch4/Scriptis_Manual.md b/web/docs/en_US/ch4/Scriptis_Manual.md new file mode 100755 index 000000000..af000bb51 --- /dev/null +++ b/web/docs/en_US/ch4/Scriptis_Manual.md @@ -0,0 +1,183 @@ +## 1 Functions + +Scriptis mainly has the following features: + +1. Workspace: Used to store scripts, data and log files. Support creating sql, hive, scala, python, pyspark scripts. +2. Dataset Module: Display datasets and tables based on user permission. +3. UDF Module: UDF are functions that can be used in sql and hql scripts. This module has the capability of managing, sharing and loading functions. +4. Function Module: Composed of personal, system and shared user-define functions. These functions can be used in python, pyspark, scala scripts. +5. HDFS Module: Personal directory of user's HDFS (distributed filesystem), used to store large files. +6. Script Module: Capable of editing, running and stopping scripts. Customizing variable configurations and shortcut keys are also supported. +7. Results: Include displaying, downloading and exporting the results. +8. Script History: Displaying the running history of the scripts. +9. Console: Users can access settings, global history, resource manager, global variables and FAQs here. +10. Bottom right pop up box: Include task, engine and queue managers. + +These functions are described in detail below. + +## 2 Workspace + +Workspace is a file directory that a user have full permission to. At here, a user could do various operations such as managing files. The recommending directory structure is: script, data, log and res, since it is quite clear and thus easy for users to check and manage. The major functions of workspace are listed below: + +1. Right-clicks on workspace, a user can select copying path, creating a directory, creating a script or refreshing. + +![ide05](../images/ch4/ide05.png) + +2. Locates on the top of this module, there is a search box for quick searching. + +![ide06](../images/ch4/ide06.png) + +3. Support creating following kinds of scripts: + +- sql: Correspond to SparkSQL in Spark engine, syntax guide: https://docs.databricks.com/spark/latest/spark-sql/index.html + +- hql: Correspond to Hive engine: syntax guide: https://cwiki.apache.org/confluence/display/Hive/LanguageManual + +- Scala: Correspond to scala in Spark engine, syntax guide: https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-scala.html + +- JDBC: sql standard syntax, not supported yet. + +- Python: Standalone python engine, compatible with python + +- PythonSpark: Correspond to python in Spark engine, syntax guide: https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html + + ![ide07](../images/ch4/ide07.png) + +4. Right-click on script folder and files under it, users can choose to rename it, delete it, open it on the right side or export to hive (csv, txt, excel files) and hdfs. + + ![ide08](../images/ch4/ide08.png) + +## 3 Dataset Module + +Dataset module has the following functions. + +1. Get the information of datasets, tables and fields. + +![ide09](../images/ch4/ide09.png) + +2. Right-click on a table and select query table option can quickly generate a temporary hive script for data lookup. + +![ide10](../images/ch4/ide10.png) + +3. Right-click on a table and select describe table option can display the detailed information of this table and its corresponding fields and partitions. + +![ide11](../images/ch4/ide11.png) + +4. Right-click on a table, select export table option can generate a corresponding csv or excel file. + +![ide12](../images/ch4/ide12.png) + +## 4 UDF Module + +This module not only makes it easy for user to classify and display UDF, but also enables users to manage and share UDF. The major functions are listed below: + +#### 4.1 Displaying and loading functions + +Default top-level directory: + +- BDAP function: Provided by platform and can be used in sql, pyspark, scala and hive (written with sql) scripts. + +- System function: Functions that system provides and loaded by default. Can be used in sql, pyspark, scala and hive (written with sql) scripts. + +- Individual function: Self-define functions, include general functions and Spark exclusive functions. + +- Sharing function: Functions created by administrator and then shared to other users. + +- Apart from system functions, other types of functions must be loaded before using and a user must kill the started session after checking the functions. + + In addition, if a function is checked and loaded, it would correspondingly shown in auto-complete options. + + ![ide13](../images/ch4/ide13.png) + +#### 4.2 Creating a new UDF + +It is quite easy to create a new UDF as long as you've finished the code. The steps are as follows: + +1. To create a general UDF, a user needs to compile the corresponding Jar package first. General means either hql in Hive or sql in Spark applies here. + +2. To create a Spark exclusive UDF, a user needs to create a corresponding python or scala script. Besides, to ensure the correctness, it is better to test the scripts. + +3. Add this UDF to Scriptis: + + General UDF: Choose general then select the path in workspace for its Jar package. Next, fill in the full class path of UDF and add formatting as well as description: + +![ide14](../images/ch4/ide14.png) + + Spark exclusive UDF -- written in scala: Check Spark then select the corresponding scala script and fill in the registration format. (Function name in script): + +![ide15](../images/ch4/ide15.png) + + Spark exclusive UDF -- written in scala: Check Spark then select the corresponding python script and fill in the registration format. (Function name in script): + +![ide16](../images/ch4/ide16.png) + +#### 4.3 Way to write Scala and Python functions: + +For a PythonUDF, a user only needs to define a function, and the scripts have to correspond with this function. + +``` +def hello(id): + return str(id) + ":hello" +``` + +The way to create a ScalaUDF is quite similar to creating a Python UDF, a user only needs to define a function: + +``` +def helloWord(str: String): String = "hello, " + str +``` + +Note: Python UDF and Scala UDF can only applied in scripts that corresponding to the Spark engine. + +## 5 Function Module + +Function module is similar to UDF module, the only difference between them is that one is UDF and the other is self-defined function. Also notes that, functions defined by python can only be used in python and pyspark. Similarly, functions defined by scala can only be used in scala. + +## 6 Script Module + +The functions of this module are mainly integrated in the script edit box: + +1. Script editing: Support basic keyword highlighting, code formatting, code merging, auto-completion and shortcuts etc. + +2. Running and stopping: Users can choose to run only a segment of code or the entire script. By clicking stop button, users can terminate the running scripts whenever they want. + +3. Script edit box has configuration options for defining user-define functions that take effects within the script. + +![ide18](../images/ch4/ide18.png) + +## 7 Results + +This module has the following functions: + +1. For now, it supports showing results in a table, clicking the header to sort, double-clicking to copy the field name and all these functions are restricted to showing up to 5000 lines of records. More functions would be supported in future, such as displaying the selected columns and filed types. + +2. Visual analysis: Click on visual analysis button to visualize the result through VSBI. (Soon to be released) + +3. Downloading: Users can directly download the results as csv and excel files to local browser. Only support downloading 5000 lines for now. + +4. Exporting: Results can be exported to the workspace (shared directory of BDAP) in either csv or excel format and would not be restricted to 5000 lines if you choose full export at first. To use full export, add a comment in front of sql: `--set wds.linkis.engine.no.limit.allow=true` + +5. Go to Console--Configuration--Pipeline--Import and Export settings--Result export type to choose whether export results in csv format or excel format. + +## 8 Script History + +Script history shows all the running information of the script. A user can quickly find the log and running results of a script that was run before and therefore, avoid running the same script repeatedly. + +![ide23](../images/ch4/ide23.png) + +## 9 Console + +Console has the following functions: + +1. Settings: Include general settings (such as setting up queues) and data development related engine settings: spark, hive, python, pipeline, etc. + +![ide25](../images/ch4/ide25.png) + +2. Global variables: A global variable is a custom variable that can be applied to all scripts. If its name is same as the name of a variable in a script, that variable would take effect. + +3. Other functions: Global history, resource manager, FAQs. + +## 10 Bottom right pop up box + +Similar to the Windows task manager, users can quickly view and manage tasks, engines and queue resources here. + +![ide24](../images/ch4/ide24.png) \ No newline at end of file diff --git a/web/docs/en_US/images/ch3/01.png b/web/docs/en_US/images/ch3/01.png new file mode 100755 index 000000000..f45529eb3 Binary files /dev/null and b/web/docs/en_US/images/ch3/01.png differ diff --git a/web/docs/en_US/images/ch3/02.png b/web/docs/en_US/images/ch3/02.png new file mode 100755 index 000000000..65c7f14cb Binary files /dev/null and b/web/docs/en_US/images/ch3/02.png differ diff --git a/web/docs/en_US/images/ch3/03.png b/web/docs/en_US/images/ch3/03.png new file mode 100755 index 000000000..b1fa65b15 Binary files /dev/null and b/web/docs/en_US/images/ch3/03.png differ diff --git a/web/docs/en_US/images/ch3/04.png b/web/docs/en_US/images/ch3/04.png new file mode 100755 index 000000000..a768f716d Binary files /dev/null and b/web/docs/en_US/images/ch3/04.png differ diff --git a/web/docs/en_US/images/ch3/05.png b/web/docs/en_US/images/ch3/05.png new file mode 100755 index 000000000..3909c97bf Binary files /dev/null and b/web/docs/en_US/images/ch3/05.png differ diff --git a/web/docs/en_US/images/ch3/06.png b/web/docs/en_US/images/ch3/06.png new file mode 100755 index 000000000..8297ce05c Binary files /dev/null and b/web/docs/en_US/images/ch3/06.png differ diff --git a/web/docs/en_US/images/ch3/07.png b/web/docs/en_US/images/ch3/07.png new file mode 100755 index 000000000..065021bfc Binary files /dev/null and b/web/docs/en_US/images/ch3/07.png differ diff --git a/web/docs/en_US/images/ch4/ide05.png b/web/docs/en_US/images/ch4/ide05.png new file mode 100755 index 000000000..10f3d91a3 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide05.png differ diff --git a/web/docs/en_US/images/ch4/ide06.png b/web/docs/en_US/images/ch4/ide06.png new file mode 100755 index 000000000..1fb5b2cd6 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide06.png differ diff --git a/web/docs/en_US/images/ch4/ide07.png b/web/docs/en_US/images/ch4/ide07.png new file mode 100755 index 000000000..fa414ab05 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide07.png differ diff --git a/web/docs/en_US/images/ch4/ide08.png b/web/docs/en_US/images/ch4/ide08.png new file mode 100755 index 000000000..5b2321f7c Binary files /dev/null and b/web/docs/en_US/images/ch4/ide08.png differ diff --git a/web/docs/en_US/images/ch4/ide09.png b/web/docs/en_US/images/ch4/ide09.png new file mode 100755 index 000000000..47dc29690 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide09.png differ diff --git a/web/docs/en_US/images/ch4/ide10.png b/web/docs/en_US/images/ch4/ide10.png new file mode 100755 index 000000000..4141452a5 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide10.png differ diff --git a/web/docs/en_US/images/ch4/ide11.png b/web/docs/en_US/images/ch4/ide11.png new file mode 100755 index 000000000..d4fb1ec8e Binary files /dev/null and b/web/docs/en_US/images/ch4/ide11.png differ diff --git a/web/docs/en_US/images/ch4/ide12.png b/web/docs/en_US/images/ch4/ide12.png new file mode 100755 index 000000000..a615495ae Binary files /dev/null and b/web/docs/en_US/images/ch4/ide12.png differ diff --git a/web/docs/en_US/images/ch4/ide13.png b/web/docs/en_US/images/ch4/ide13.png new file mode 100755 index 000000000..ff4b6d3c0 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide13.png differ diff --git a/web/docs/en_US/images/ch4/ide14.png b/web/docs/en_US/images/ch4/ide14.png new file mode 100755 index 000000000..9ca07e562 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide14.png differ diff --git a/web/docs/en_US/images/ch4/ide15.png b/web/docs/en_US/images/ch4/ide15.png new file mode 100755 index 000000000..740027f13 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide15.png differ diff --git a/web/docs/en_US/images/ch4/ide16.png b/web/docs/en_US/images/ch4/ide16.png new file mode 100755 index 000000000..a4a11ada2 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide16.png differ diff --git a/web/docs/en_US/images/ch4/ide17.png b/web/docs/en_US/images/ch4/ide17.png new file mode 100755 index 000000000..02576e3f9 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide17.png differ diff --git a/web/docs/en_US/images/ch4/ide18.png b/web/docs/en_US/images/ch4/ide18.png new file mode 100755 index 000000000..a69059d21 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide18.png differ diff --git a/web/docs/en_US/images/ch4/ide19.png b/web/docs/en_US/images/ch4/ide19.png new file mode 100755 index 000000000..49359d546 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide19.png differ diff --git a/web/docs/en_US/images/ch4/ide20.png b/web/docs/en_US/images/ch4/ide20.png new file mode 100755 index 000000000..12b7caf0f Binary files /dev/null and b/web/docs/en_US/images/ch4/ide20.png differ diff --git a/web/docs/en_US/images/ch4/ide21.png b/web/docs/en_US/images/ch4/ide21.png new file mode 100755 index 000000000..74bafffc2 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide21.png differ diff --git a/web/docs/en_US/images/ch4/ide22.png b/web/docs/en_US/images/ch4/ide22.png new file mode 100755 index 000000000..a767e9a21 Binary files /dev/null and b/web/docs/en_US/images/ch4/ide22.png differ diff --git a/web/docs/en_US/images/ch4/ide23.png b/web/docs/en_US/images/ch4/ide23.png new file mode 100755 index 000000000..bfb5eef3d Binary files /dev/null and b/web/docs/en_US/images/ch4/ide23.png differ diff --git a/web/docs/en_US/images/ch4/ide24.png b/web/docs/en_US/images/ch4/ide24.png new file mode 100755 index 000000000..19c922ecc Binary files /dev/null and b/web/docs/en_US/images/ch4/ide24.png differ diff --git a/web/docs/en_US/images/ch4/ide25.png b/web/docs/en_US/images/ch4/ide25.png new file mode 100755 index 000000000..be473089c Binary files /dev/null and b/web/docs/en_US/images/ch4/ide25.png differ diff --git a/web/docs/en_US/images/readme/Comparison.png b/web/docs/en_US/images/readme/Comparison.png new file mode 100755 index 000000000..df3144cc6 Binary files /dev/null and b/web/docs/en_US/images/readme/Comparison.png differ diff --git a/web/docs/en_US/images/readme/cs.gif b/web/docs/en_US/images/readme/cs.gif new file mode 100755 index 000000000..d735e1ba6 Binary files /dev/null and b/web/docs/en_US/images/readme/cs.gif differ diff --git a/web/docs/en_US/images/readme/datasource.gif b/web/docs/en_US/images/readme/datasource.gif new file mode 100755 index 000000000..555a208e3 Binary files /dev/null and b/web/docs/en_US/images/readme/datasource.gif differ diff --git a/web/docs/en_US/images/readme/results.gif b/web/docs/en_US/images/readme/results.gif new file mode 100755 index 000000000..cfef7cebb Binary files /dev/null and b/web/docs/en_US/images/readme/results.gif differ diff --git a/web/docs/en_US/images/readme/running.gif b/web/docs/en_US/images/readme/running.gif new file mode 100755 index 000000000..9d310bc88 Binary files /dev/null and b/web/docs/en_US/images/readme/running.gif differ diff --git a/web/docs/en_US/images/readme/script.gif b/web/docs/en_US/images/readme/script.gif new file mode 100755 index 000000000..39b729d0a Binary files /dev/null and b/web/docs/en_US/images/readme/script.gif differ diff --git a/web/docs/en_US/images/readme/setup.gif b/web/docs/en_US/images/readme/setup.gif new file mode 100755 index 000000000..7515448fb Binary files /dev/null and b/web/docs/en_US/images/readme/setup.gif differ diff --git "a/web/docs/en_US/images/readme/\345\257\271\346\257\224.png" "b/web/docs/en_US/images/readme/\345\257\271\346\257\224.png" new file mode 100755 index 000000000..d32188e5f Binary files /dev/null and "b/web/docs/en_US/images/readme/\345\257\271\346\257\224.png" differ diff --git a/web/docs/en_US/images/wechat.jpg b/web/docs/en_US/images/wechat.jpg new file mode 100755 index 000000000..1e0e913e7 Binary files /dev/null and b/web/docs/en_US/images/wechat.jpg differ diff --git a/web/docs/en_US/images/wechatQQ.png b/web/docs/en_US/images/wechatQQ.png new file mode 100755 index 000000000..01ec041d7 Binary files /dev/null and b/web/docs/en_US/images/wechatQQ.png differ diff --git a/web/docs/zh_CN/README.md b/web/docs/zh_CN/README.md new file mode 100755 index 000000000..5567745a3 --- /dev/null +++ b/web/docs/zh_CN/README.md @@ -0,0 +1,56 @@ +Scriptis +============ + +[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) + +[English](../../README.md) | Chinese + +## 引言： + +Scriptis是一款支持在线写SQL、Pyspark、HiveQL等脚本，提交给**Linkis**（Linkis是什么？[点我了解](https://github.com/WeBankFinTech/Linkis/blob/master/docs/zh_CN/README.md)）执行的数据分析Web工具，且支持UDF、函数、资源管控和智能诊断等企业级特性。 + +---- + +## 核心特点： + +**脚本编辑**：多语言、自动补全、语法高亮、SQL语法纠错； + +![脚本编辑](images/readme/script.gif) + +**计算引擎**：基于Linkis对接多计算引擎：Spark、Hive、TiSpark等； + + +**运行时功能**：Job全生命周期展示、错误码； + +![运行时功能](images/readme/running.gif) + +**上下文**：UDF和函数管理、函数共享、自定义变量； + +![上下文](images/readme/cs.gif) + +**结果集**：多结果集支持、结果集别名自定义、结果集一键发布到报表系统； + +![结果集](images/readme/results.gif) + +**数据库**：数据库管理功能、文件（CSV/Excel）快速导入导出到表； + +![数据库](images/readme/datasource.gif) + +**管理台**：多租户资源管控、引擎参数个性化配置、任务和会话管理。 + +![管理台](images/readme/setup.gif) +---- + +### 同类系统对比： +![对比图](images/readme/对比.png) + +## 社区 +如果您想得到最快的响应，请给我们提issue，或者您也可以扫码进群： + +![WeChatQQ](/docs/en_US/images/wechatQQ.png) + +### 文档： +* [部署手册](ch1/前台部署文档.md) +* [前台编译手册](ch2/编译文档.md) +* [快速使用手册](ch3/scriptis快速使用文档.md) +* [使用手册](ch4/Scriptis使用手册.md) diff --git a/web/docs/zh_CN/SUMMARY.md b/web/docs/zh_CN/SUMMARY.md new file mode 100755 index 000000000..b19f779aa --- /dev/null +++ b/web/docs/zh_CN/SUMMARY.md @@ -0,0 +1,11 @@ +# Summary + +* [Introduction](README.md) +* [第一章 前台部署]() + * [前台部署手册](ch1/前台部署文档.md) +* [第二章 前台编译 ]() + * [前台编译手册](ch2/编译文档.md) +* [第三章 快速使用手册 ]() + * [scriptis快速使用手册](ch3/scriptis快速使用文档.md) +* [第四章 使用手册 ]() + * [scriptis快速使用手册](ch4/Scriptis使用手册.md) \ No newline at end of file diff --git "a/web/docs/zh_CN/ch1/\345\211\215\345\217\260\351\203\250\347\275\262\346\226\207\346\241\243.md" "b/web/docs/zh_CN/ch1/\345\211\215\345\217\260\351\203\250\347\275\262\346\226\207\346\241\243.md" new file mode 100755 index 000000000..cd5a68d5d --- /dev/null +++ "b/web/docs/zh_CN/ch1/\345\211\215\345\217\260\351\203\250\347\275\262\346\226\207\346\241\243.md" @@ -0,0 +1,109 @@ +Scriptis是基于Linkis开发的数据分析工具，部署Scriptis前需要先将Linkis进行部署，Linkis的部署手册见：[Linkis部署手册](https://github.com/WeBankFinTech/Linkis/blob/master/docs/zh_CN/ch1/deploy.md) + +## 1、准备工作 + +1. 点击release 选择对应的安装包进行下载 +2. 将下载下来的安装包在安装目录进行解压：unzip wedatasphere-scriptis-0.7.0-dist.zip + +## 2、部署 +    分为两种部署方式，自动化部署和手动部署 + +### 2.1 自动化部署 +    进入前端目录```wedatasphere-scriptis``` 在该目录下编辑 ```vi config.sh ``` +更改前端端口和后端接口地址,后端接口地址为linkis的gateway地址 +```$xslt +# Configuring front-end ports +scriptis_port="8088" + +# URL of the backend linkis gateway +linkis_url="http://localhost:20401" + +# Scriptis ip address +scriptis_ipaddr=$(ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}') +``` + +修改完后在该目录下执行：```sudo sh install.sh > install.log 2>&1``` + +执行完后可以直接通过在谷歌浏览器访问：```http://scriptis_ipaddr:scriptis_port``` 其中scriptis_port为config.sh里面配置的端口，scriptis_ipaddr为安装机器的IP + +如果访问失败：可以通过查看 install.log的日志查看哪一步出错 + +### 2.2 手动部署 +1.安装Nginx:```sudo yum install nginx -y``` + +2.修改配置文件：sudo vi /etc/nginx/conf.d/scriptis.conf +添加如下内容： +``` +server { + listen 8080;# 访问端口 + server_name localhost; + #charset koi8-r; + #access_log /var/log/nginx/host.access.log main; + location / { + root /appcom/Install/scriptis/ROOT; # 前端包解压的目录 + index index.html index.html; + } + location /ws {#webSocket配置支持 + proxy_pass http://192.168.xxx.xxx:9001;#linkis-gateway服务的ip端口 + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + location /api { + proxy_pass http://192.168.xxx.xxx:9001; # linkis-gateway服务的ip端口 + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header x_real_ipP $remote_addr; + proxy_set_header remote_addr $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_connect_timeout 4s; + proxy_read_timeout 600s; + proxy_send_timeout 12s; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection upgrade; + } + #error_page 404 /404.html; + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } + } + +``` + +3.将前端包拷贝到对应的目录:```/appcom/Install/scriptis/ROOT; # 前端包解压的目录 ``` + +4.启动服务```sudo systemctl restart nginx``` + +5.执行完后可以直接通过在谷歌浏览器访问：```http://nginx_ip:nginx_port``` + +## 3、常见问题 + +(1)上传文件大小限制 + +``` +sudo vi /etc/nginx/nginx.conf +``` + +更改上传大小 + +``` +client_max_body_size 200m +``` + + (2)接口超时 + +``` +sudo vi /etc/nginx/conf.d/scriptis.conf +``` + + +更改接口超时时间 + +``` +proxy_read_timeout 600s +``` + diff --git "a/web/docs/zh_CN/ch2/\347\274\226\350\257\221\346\226\207\346\241\243.md" "b/web/docs/zh_CN/ch2/\347\274\226\350\257\221\346\226\207\346\241\243.md" new file mode 100755 index 000000000..ee447703a --- /dev/null +++ "b/web/docs/zh_CN/ch2/\347\274\226\350\257\221\346\226\207\346\241\243.md" @@ -0,0 +1,86 @@ +# 编译文档中文版 + +## 启动流程 + +### 一、安装Node.js +将Node.js下载到电脑本地，安装即可。下载地址：[http://nodejs.cn/download/](http://nodejs.cn/download/) （建议使用最新的稳定版本） +**该步骤仅第一次使用时需要执行。** + +### 二、安装项目 +在终端命令行中执行以下指令： + +``` +git clone ${ipAddress} +cd DataSphereStudio/web +npm install +``` + +指令简介： +1. 将项目包从远程仓库拉取到电脑本地：git clone ${ipAddress} +2. 进入项目包根目录：cd DataSphereStudio/web +3. 安装项目所需依赖：npm install + +**该步骤仅第一次使用时需要执行。** + +### 三、配置 +您需要在代码中进行一些配置，如后端接口地址，后端socket地址等，如根目录下的.env.development文件： + +``` +// 后端接口地址 +VUE_APP_MN_CONFIG_PREFIX=http://yourIp:yourPort/yourPath +// 后端socket地址 +VUE_APP_MN_CONFIG_SOCKET=/yourSocketPath +``` + +配置的具体解释可参考vue-cli官方文档：[环境变量和模式](https://cli.vuejs.org/zh/guide/mode-and-env.html#%E7%8E%AF%E5%A2%83%E5%8F%98%E9%87%8F%E5%92%8C%E6%A8%A1%E5%BC%8F) + +### 打包项目 +您可以通过在终端命令行执行以下指令对项目进行打包，生成压缩后的代码： + +``` +npm run build +``` + +该指令成功执行后，项目根目录下会出现一个名叫 “dist” 的文件夹，该文件夹即为打包好的代码。您可以直接将该文件夹放进您的静态服务器中。 + +### 运行项目 +如果您想在本地浏览器上运行该项目并且改动代码查看效果，需要在终端命令行中执行以下指令： + +``` +npm run serve +``` + +在浏览器中（建议Chrome浏览器）通过链接访问应用：[http://localhost:8080/](http://localhost:8080/) . +当您使用该方式运行项目时，您对代码的改动产生的效果，会动态体现在浏览器上。 + +**注意：因为项目采用前后端分离开发，所以在本地浏览器上运行时，需要对浏览器进行设置跨域才能访问后端接口：** + +比如chrome浏览器： +windows系统下的配置方式： +1. 关闭所有的chrome浏览器。 +2. 新建一个chrome快捷方式，右键“属性”，“快捷方式”选项卡里选择“目标”，添加  --args --disable-web-security --user-data-dir=C:\MyChromeDevUserData +3. 通过快捷方式打开chrome浏览器 +mac系统下的配置方式： +在终端命令行执行以下命令(需要替换路径中的yourname，若还不生效请检查您机器上MyChromeDevUserData文件夹的位置并将路径复制到下面指令的“--user-data-dir=”后面) + +``` +open -n /Applications/Google\ Chrome.app/ --args --disable-web-security --user-data-dir=/Users/yourname/MyChromeDevUserData/ +``` + + +### 常见问题 + +#### npm install无法成功 +如果遇到该情况，可以使用国内的淘宝npm镜像： + +``` +npm install -g cnpm --registry=https://registry.npm.taobao.org +``` + +接着，通过执行以下指令代替npm install指令 + +``` +cnpm install +``` + +注意，项目启动和打包时，仍然可以使用npm run build和npm run serve指令 \ No newline at end of file diff --git "a/web/docs/zh_CN/ch3/scriptis\345\277\253\351\200\237\344\275\277\347\224\250\346\226\207\346\241\243.md" "b/web/docs/zh_CN/ch3/scriptis\345\277\253\351\200\237\344\275\277\347\224\250\346\226\207\346\241\243.md" new file mode 100755 index 000000000..f8aff5f66 --- /dev/null +++ "b/web/docs/zh_CN/ch3/scriptis\345\277\253\351\200\237\344\275\277\347\224\250\346\226\207\346\241\243.md" @@ -0,0 +1,33 @@ +Scriptis快速使用文档 +============ + +## 快速登录 +     为了方便用户使用，系统默认通过使用Linkis的部署用户名进行登录，比如是hadoop部署的可以直接通过 用户：hadoop，密码：hadoop(密码就是用户名)来进行登录。 +首先输入前端容器地址：192.168.xx.xx:8888 接着输入用户名密码：hadoop/hadoop +![01](../images/ch3/01.png) +__备注：__ 其他用户登录必须使用配置的LDAP的密码进行登录 + +## 设置队列 +     scriptis的Spark引擎是通过开发-->设置-->yarn队列名-->输入有权限的yarn 队列名 +![02](../images/ch3/02.png) + +## 创建脚本 +     设置完队列后，就可以建立一个sql脚本使用SparkSQL进行数据查询，用户只需要输入sql。点击工作空间-个人目录-右键创建sql脚本 + +1. 创建 +![03](../images/ch3/03.png) +2. 建立sql脚本 +![04](../images/ch3/04.png) +3. 输入sql脚本 +![05](../images/ch3/05.png) + +## 点击运行 +     点击脚本的运行按钮则可以运行脚本，并可以通过日志获悉运行状况 +![01](../images/ch3/06.png) + +## 结果展示 +     点击运行后，等待一段时间，第一次运行可能会稍微久点，因为需要启动引擎。运行完成后会将结果进行展示，用户可以对结果进行下载，导出等操作 +![01](../images/ch3/07.png) + +__详细的使用手册见：__ [使用手册文档](../ch4/Scriptis使用手册.md) + diff --git "a/web/docs/zh_CN/ch4/Scriptis\344\275\277\347\224\250\346\211\213\345\206\214.md" "b/web/docs/zh_CN/ch4/Scriptis\344\275\277\347\224\250\346\211\213\345\206\214.md" new file mode 100755 index 000000000..05c5df60a --- /dev/null +++ "b/web/docs/zh_CN/ch4/Scriptis\344\275\277\347\224\250\346\211\213\345\206\214.md" @@ -0,0 +1,130 @@ +## 1 功能简介 + + +scirptis主要拆分为一下功能： +1. 工作空间：用户存储用户的脚本，数据，日志等文件；支持建立：sql，hive，scala，python，pyspark等类型的脚本； +2. 数据库模块：展示用户拥有权限的数据库和表信息，以及表导入导出； +3. UDF模块：UDF是可以在sql和hql脚本中使用的函数，包括函数管理，勾选加载，共享等； +4. 方法模块：方法是用户自己，系统，共享的自定义函数，支持在python，pyspark，scala脚本中使用； +5. HDFS模块：用户的HDFS（分布式文件系统）个人目录，用于存放大数据文件； +6. 脚本模块：包括脚本编辑，运行，停止，自定义变量配置，快捷键； +7. 运行结果：包括结果展示，下载，导出； +8. 脚本历史：脚本历史展示了该脚本历史运行信息； +9. 管理台：主要包含设置，全局历史，资源管理器，全局变量，常见问题； +10. 右下角弹出框：包含任务管理器，引擎管理器，队列管理器； +下面对这些功能进行详细介绍。 + +## 2 工作空间 + +工作空间是一个文件目录，用户对该目录拥有所有的权限可以进行文件管理操作等。建议的目录结构是：script,data,log,res四个目录，目录结构清晰方便用户进行查看和管理。工作空间主要功能如下： +1. 工作空间右键主要包含复制路径，新建目录，新建脚本，刷新 +![ide05](../images/ch4/ide05.png) +2. 顶上搜索功能，支持对文件快速定位 +![ide06](../images/ch4/ide06.png) +3. 新建脚本功能支持建立以下脚本 + +* sql:对应Spark引擎的SparkSQL，语法指导手册：[sql语法指导手册](https://docs.databricks.com/spark/latest/spark-sql/index.html) +* hql：对应的是Hive引擎，Hive语法指导手册：[Hive语法指导手册](https://cwiki.apache.org/confluence/display/Hive/LanguageManual) +* Scala：对应的是Spark引擎的Scala语法，语法指导手册：[Spark Scala指导手册](https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-scala.html) +* Python：单机Python引擎，兼容Python语法 +* PythonSpark：对应的是Spark引擎的Python语法，语法指导手册：[pySpark指导手册](https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html) + ![ide07](../images/ch4/ide07.png) +* 脚本右键功能，脚本右键主要有打卡到侧边，复制路径，重命名，删除，导入到hive（csv,txt,excel类型文件），导入到hdfs等功能 + ![ide08](../images/ch4/ide08.png) + +## 3 数据库模块 + +数据库模块主要有以下功能： +1. 刷库，刷表，刷字段信息 +![ide09](../images/ch4/ide09.png) +2. 表右键功能--查询表：快捷生产临时hive脚本进行数据查看 +![ide10](../images/ch4/ide10.png) +3. 表右键功能--查看表结构：展示表的字段详细信息，表详情信息，表分区信息等： +![ide11](../images/ch4/ide11.png) +4. 表右键功能--导出表：可以导出为csv和excel等文件 +![ide12](../images/ch4/ide12.png) + +## 4 UDF模块 + +UDF功能是方便用户对UDF进行分类展示，以及用户可以对个人函数进行管理，共享等。主要包含以下功能： + +#### 4.1UDF展示和加载功能 + +默认顶层目录是： + +* 平台函数：平台提供的UDF函数，可以在sql，pysaprk，scala，hive脚本的sql语言里面进行使用 +* 系统函数：系统默认已经提供并自动加载了的函数，可以在sql，pysaprk，scala，hive脚本的sql语言里面进行使用 +* 个人函数：个人自己新建立的UDF，包含通用函数，Spark专用函数。 +* 共享函数：科室管理员建立的UDF函数，并共享给其他人使用 + 除了系统函数外，其他类别的函数都必须点击加载后才能进行使用，而且点击勾选后一定要先kill掉已经启动的会话。 + 另外勾选加载的UDF，在脚本中也会进行自动补全提示： + ![ide13](../images/ch4/ide13.png) + +#### 4.2UDF新增函数功能： + +创建一个UDF很简单，主要在于做好前期的UDF代码的准备。步骤主要如下： +1. 建立通用UDF，需要先编译好相应的Jar包。这里的通用是指Hive的hql和Spark的sql都可以用 +2. 建立Spark专用的UDF，需要先建立好相应的py脚本或者scala脚本，为了使用的正确性，最好测试下相应的脚本 +3. 在开发建立该UDF: +通用UDF：需要勾选通用，并选择对应的jar包路径，填写UDF的全类路径，填写使用格式和描述： +![ide14](../images/ch4/ide14.png) +Spark专用--scala脚本：需要勾选Spark，并选择对应的scala脚本，填写注册格式（填写相应脚本中的函数名）： +![ide15](../images/ch4/ide15.png) +Spark专用--Python脚本：需要勾选Spark，并选择对应的python脚本，填写注册格式（填写相应脚本中的函数名）： +![ide16](../images/ch4/ide16.png) + +#### 4.3Scala和Python函数写法： + +PythonUDF建立方式，只需要定义一个方法即可，对应的脚本只能与该方法有关的内容： + +``` +def hello(id): + return str(id) + ":hello" +``` + +ScalaUDF 建立方式和Python类似，定义一个方法即可： + +``` +def helloWorld(str: String): String = "hello, " + str +``` + +注意：PythonUDF和ScalaUDF只能在Spark引擎对应的脚本中使用 + +## 5 方法模块： + +方法模块和UDF模块功能类似，只是一个是UDF一个自定义函数。另外需要注意的是：python脚本定义的函数只能在python和pysaprk中使用，scala脚本定义的函数只能在scala中进行使用。 + +## 6 脚本模块 + +脚本模块就是脚本编辑框包含的功能，主要有： +1. 脚本编辑：基础的关键字高亮功能，代码格式化，代码合并，自动补全，快捷健等功能。 +2. 运行和停止：用户可以选择代码进行执行，也可以直接点击执行运行整个脚本。脚本运行起来后可以通过停止按钮进行停止。 +3. 脚本编辑框有配置选项用于定义用户的自定义变量，该脚本内生效 +![ide18](../images/ch4/ide18.png) + +## 7 运行结果 + +运行结果包括以下功能： +1. 表格结果展示，点击表头可以进行排序，双击复制字段名等，功能会继续丰富，包括选择列展示，展示字段类型。只支持展示5000行 +2. 可视化分析：点击可视化分析，可以通过VSBI对产生的结果集进行可视化展示（待开源） +3. 下载：结果集支持直接下载到浏览器本地，支持CSV和excel，只支持下载5000行 +4. 导出：结果集导出可以支持导出到工作空间（BDAP的共享目录），结果集导出也支持CSV和Excel格式，并且通过打开全量导出，可以超过5000行的限制。全量导出开关，在执行的sql前面加上注释：`--set wds.linkis.engine.no.limit.allow=true` + +## 8 脚本历史 + +脚本历史展示了该脚本文件的历史运行信息，可以快速找到之前运行的脚本的日志，结果。可以有效减少多次运行相同的脚本： +![ide23](../images/ch4/ide23.png) + +## 9 管理台 + +管理台主要包含一下功能： +1. 设置：包含通用设置（如设置队列），数据开发相关引擎的设置：spark，hive，python，pipeline等 +![ide25](../images/ch4/ide25.png) +2. 全局变量：全局变量是可以用于所有脚本的自定义变量，如果变量重名则脚本中配置的生效。 +3. 其他功能：全局历史，资源管理器，常见问题 + +## 10 右下角弹出框 + +右下角弹出框点击后是类似于Windows的任务管理器类似的功能，用户可以快速的对任务管，引擎，队列资源进行查看和管理： +![ide24](../images/ch4/ide24.png) + diff --git a/web/docs/zh_CN/images/ch3/01.png b/web/docs/zh_CN/images/ch3/01.png new file mode 100755 index 000000000..f45529eb3 Binary files /dev/null and b/web/docs/zh_CN/images/ch3/01.png differ diff --git a/web/docs/zh_CN/images/ch3/02.png b/web/docs/zh_CN/images/ch3/02.png new file mode 100755 index 000000000..65c7f14cb Binary files /dev/null and b/web/docs/zh_CN/images/ch3/02.png differ diff --git a/web/docs/zh_CN/images/ch3/03.png b/web/docs/zh_CN/images/ch3/03.png new file mode 100755 index 000000000..b1fa65b15 Binary files /dev/null and b/web/docs/zh_CN/images/ch3/03.png differ diff --git a/web/docs/zh_CN/images/ch3/04.png b/web/docs/zh_CN/images/ch3/04.png new file mode 100755 index 000000000..a768f716d Binary files /dev/null and b/web/docs/zh_CN/images/ch3/04.png differ diff --git a/web/docs/zh_CN/images/ch3/05.png b/web/docs/zh_CN/images/ch3/05.png new file mode 100755 index 000000000..3909c97bf Binary files /dev/null and b/web/docs/zh_CN/images/ch3/05.png differ diff --git a/web/docs/zh_CN/images/ch3/06.png b/web/docs/zh_CN/images/ch3/06.png new file mode 100755 index 000000000..8297ce05c Binary files /dev/null and b/web/docs/zh_CN/images/ch3/06.png differ diff --git a/web/docs/zh_CN/images/ch3/07.png b/web/docs/zh_CN/images/ch3/07.png new file mode 100755 index 000000000..065021bfc Binary files /dev/null and b/web/docs/zh_CN/images/ch3/07.png differ diff --git a/web/docs/zh_CN/images/ch4/ide05.png b/web/docs/zh_CN/images/ch4/ide05.png new file mode 100755 index 000000000..10f3d91a3 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide05.png differ diff --git a/web/docs/zh_CN/images/ch4/ide06.png b/web/docs/zh_CN/images/ch4/ide06.png new file mode 100755 index 000000000..1fb5b2cd6 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide06.png differ diff --git a/web/docs/zh_CN/images/ch4/ide07.png b/web/docs/zh_CN/images/ch4/ide07.png new file mode 100755 index 000000000..fa414ab05 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide07.png differ diff --git a/web/docs/zh_CN/images/ch4/ide08.png b/web/docs/zh_CN/images/ch4/ide08.png new file mode 100755 index 000000000..5b2321f7c Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide08.png differ diff --git a/web/docs/zh_CN/images/ch4/ide09.png b/web/docs/zh_CN/images/ch4/ide09.png new file mode 100755 index 000000000..47dc29690 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide09.png differ diff --git a/web/docs/zh_CN/images/ch4/ide10.png b/web/docs/zh_CN/images/ch4/ide10.png new file mode 100755 index 000000000..4141452a5 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide10.png differ diff --git a/web/docs/zh_CN/images/ch4/ide11.png b/web/docs/zh_CN/images/ch4/ide11.png new file mode 100755 index 000000000..d4fb1ec8e Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide11.png differ diff --git a/web/docs/zh_CN/images/ch4/ide12.png b/web/docs/zh_CN/images/ch4/ide12.png new file mode 100755 index 000000000..a615495ae Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide12.png differ diff --git a/web/docs/zh_CN/images/ch4/ide13.png b/web/docs/zh_CN/images/ch4/ide13.png new file mode 100755 index 000000000..ff4b6d3c0 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide13.png differ diff --git a/web/docs/zh_CN/images/ch4/ide14.png b/web/docs/zh_CN/images/ch4/ide14.png new file mode 100755 index 000000000..9ca07e562 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide14.png differ diff --git a/web/docs/zh_CN/images/ch4/ide15.png b/web/docs/zh_CN/images/ch4/ide15.png new file mode 100755 index 000000000..740027f13 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide15.png differ diff --git a/web/docs/zh_CN/images/ch4/ide16.png b/web/docs/zh_CN/images/ch4/ide16.png new file mode 100755 index 000000000..a4a11ada2 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide16.png differ diff --git a/web/docs/zh_CN/images/ch4/ide17.png b/web/docs/zh_CN/images/ch4/ide17.png new file mode 100755 index 000000000..02576e3f9 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide17.png differ diff --git a/web/docs/zh_CN/images/ch4/ide18.png b/web/docs/zh_CN/images/ch4/ide18.png new file mode 100755 index 000000000..a69059d21 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide18.png differ diff --git a/web/docs/zh_CN/images/ch4/ide19.png b/web/docs/zh_CN/images/ch4/ide19.png new file mode 100755 index 000000000..49359d546 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide19.png differ diff --git a/web/docs/zh_CN/images/ch4/ide20.png b/web/docs/zh_CN/images/ch4/ide20.png new file mode 100755 index 000000000..12b7caf0f Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide20.png differ diff --git a/web/docs/zh_CN/images/ch4/ide21.png b/web/docs/zh_CN/images/ch4/ide21.png new file mode 100755 index 000000000..74bafffc2 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide21.png differ diff --git a/web/docs/zh_CN/images/ch4/ide22.png b/web/docs/zh_CN/images/ch4/ide22.png new file mode 100755 index 000000000..a767e9a21 Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide22.png differ diff --git a/web/docs/zh_CN/images/ch4/ide23.png b/web/docs/zh_CN/images/ch4/ide23.png new file mode 100755 index 000000000..bfb5eef3d Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide23.png differ diff --git a/web/docs/zh_CN/images/ch4/ide24.png b/web/docs/zh_CN/images/ch4/ide24.png new file mode 100755 index 000000000..19c922ecc Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide24.png differ diff --git a/web/docs/zh_CN/images/ch4/ide25.png b/web/docs/zh_CN/images/ch4/ide25.png new file mode 100755 index 000000000..be473089c Binary files /dev/null and b/web/docs/zh_CN/images/ch4/ide25.png differ diff --git a/web/docs/zh_CN/images/readme/cs.gif b/web/docs/zh_CN/images/readme/cs.gif new file mode 100755 index 000000000..4df76a164 Binary files /dev/null and b/web/docs/zh_CN/images/readme/cs.gif differ diff --git a/web/docs/zh_CN/images/readme/datasource.gif b/web/docs/zh_CN/images/readme/datasource.gif new file mode 100755 index 000000000..eedea96f5 Binary files /dev/null and b/web/docs/zh_CN/images/readme/datasource.gif differ diff --git a/web/docs/zh_CN/images/readme/results.gif b/web/docs/zh_CN/images/readme/results.gif new file mode 100755 index 000000000..c95943c95 Binary files /dev/null and b/web/docs/zh_CN/images/readme/results.gif differ diff --git a/web/docs/zh_CN/images/readme/running.gif b/web/docs/zh_CN/images/readme/running.gif new file mode 100755 index 000000000..81ff07240 Binary files /dev/null and b/web/docs/zh_CN/images/readme/running.gif differ diff --git a/web/docs/zh_CN/images/readme/script.gif b/web/docs/zh_CN/images/readme/script.gif new file mode 100755 index 000000000..a503694bd Binary files /dev/null and b/web/docs/zh_CN/images/readme/script.gif differ diff --git a/web/docs/zh_CN/images/readme/setup.gif b/web/docs/zh_CN/images/readme/setup.gif new file mode 100755 index 000000000..dabb635f8 Binary files /dev/null and b/web/docs/zh_CN/images/readme/setup.gif differ diff --git "a/web/docs/zh_CN/images/readme/\345\257\271\346\257\224.png" "b/web/docs/zh_CN/images/readme/\345\257\271\346\257\224.png" new file mode 100755 index 000000000..d32188e5f Binary files /dev/null and "b/web/docs/zh_CN/images/readme/\345\257\271\346\257\224.png" differ diff --git a/web/install.sh b/web/install.sh old mode 100644 new mode 100755 index 9a5a410c6..6403d5d1b --- a/web/install.sh +++ b/web/install.sh @@ -1,22 +1,12 @@ #!/bin/bash #当前路径 -shellDir=`dirname $0` -workDir=`cd ${shellDir}/..;pwd` - -echo "dss web install start" - -dss_web_port=$DSS_WEB_PORT -linkis_gateway_url=$LINKIS_GATEWAY_URL -dss_nginx_ip=$DSS_NGINX_IP -#linkis_eureka_url=$LINKIS_EUREKA_URL - -if [[ "$dss_web_port" == "" ]]; then -source ${workDir}/conf/config.sh -fi +workDir=$(cd `dirname $0`; pwd) +echo "dss front-end deployment script" +source $workDir/config.sh # 前端放置目录，默认为解压目录 dss_basepath=$workDir @@ -54,11 +44,11 @@ version=`cat /etc/redhat-release|sed -r 's/.* ([0-9]+)\..*/\1/'` echo "========================================================================配置信息=======================================================================" -echo "DSS web访问端口：${dss_web_port}" -echo "Linkis gateway 的地址：${linkis_gateway_url}" -echo "DSS web 静态文件地址：${dss_basepath}/dist" -echo "DSS web 安装路径：${workDir}" -echo "DSS nginx ip：${dss_nginx_ip}" +echo "前端访问端口：${dss_port}" +echo "后端Linkis的地址：${linkis_url}" +echo "静态文件地址：${dss_basepath}/dist" +echo "当前路径：${workDir}" +echo "本机ip：${dss_ipaddr}" echo "========================================================================配置信息=======================================================================" echo "" @@ -66,33 +56,42 @@ echo "" # 创建文件并配置nginx dssConf(){ - s_host='$host' + + s_host='$host' s_remote_addr='$remote_addr' s_proxy_add_x_forwarded_for='$proxy_add_x_forwarded_for' s_http_upgrade='$http_upgrade' - sudo sh -c "echo ' + echo " server { - listen $dss_web_port;# 访问端口 + listen $dss_port;# 访问端口 server_name localhost; #charset koi8-r; #access_log /var/log/nginx/host.access.log main; + location /dss/visualis { root ${dss_basepath}; # 静态文件目录 autoindex on; } + + location /dss/linkis { + root ${dss_basepath}; # linkis管理台的静态文件目录 + autoindex on; + } + location / { root ${dss_basepath}/dist; # 静态文件目录 index index.html index.html; } + location /ws { - proxy_pass $linkis_gateway_url;#后端Linkis的地址 + proxy_pass $linkis_url;#后端Linkis的地址 proxy_http_version 1.1; proxy_set_header Upgrade $s_http_upgrade; proxy_set_header Connection "upgrade"; } location /api { - proxy_pass $linkis_gateway_url; #后端Linkis的地址 + proxy_pass $linkis_url; #后端Linkis的地址 proxy_set_header Host $s_host; proxy_set_header X-Real-IP $s_remote_addr; proxy_set_header x_real_ipP $s_remote_addr; @@ -114,7 +113,7 @@ dssConf(){ root /usr/share/nginx/html; } } - ' > /etc/nginx/conf.d/dss.conf" + " > /etc/nginx/conf.d/dss.conf } @@ -123,28 +122,28 @@ centos7(){ # nginx是否安装 #sudo rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm sudo yum install -y nginx - sudo echo "nginx 安装成功" + echo "nginx 安装成功" # 配置nginx dssConf # 解决 0.0.0.0:8888 问题 - sudo yum -y install policycoreutils-python - sudo semanage port -a -t http_port_t -p tcp $dss_web_port + yum -y install policycoreutils-python + semanage port -a -t http_port_t -p tcp $dss_port # 开放前端访问端口 - sudo firewall-cmd --zone=public --add-port=$dss_web_port/tcp --permanent + firewall-cmd --zone=public --add-port=$dss_port/tcp --permanent # 重启防火墙 - sudo firewall-cmd --reload + firewall-cmd --reload # 启动nginx - sudo systemctl restart nginx + systemctl restart nginx # 调整SELinux的参数 - sudo sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config + sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config # 临时生效 - sudo setenforce 0 + setenforce 0 } @@ -153,7 +152,7 @@ centos6(){ # yum S_basearch='$basearch' S_releasever='$releasever' - sudo echo " + echo " [nginx] name=nginx repo baseurl=http://nginx.org/packages/centos/$E_releasever/$S_basearch/ @@ -162,31 +161,31 @@ centos6(){ " >> /etc/yum.repos.d/nginx.repo # install nginx - sudo yum install nginx -y + yum install nginx -y # 配置nginx dssConf # 防火墙 - S_iptables=`lsof -i:$dss_web_port | wc -l` + S_iptables=`lsof -i:$dss_port | wc -l` if [ "$S_iptables" -gt "0" ];then # 已开启端口防火墙重启 - sudo service iptables restart + service iptables restart else # 未开启防火墙添加端口再重启 - sudo iptables -I INPUT 5 -i eth0 -p tcp --dport $dss_web_port -m state --state NEW,ESTABLISHED -j ACCEPT - sudo service iptables save - sudo service iptables restart + iptables -I INPUT 5 -i eth0 -p tcp --dport $dss_port -m state --state NEW,ESTABLISHED -j ACCEPT + service iptables save + service iptables restart fi # start - sudo /etc/init.d/nginx start + /etc/init.d/nginx start # 调整SELinux的参数 - sudo sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config + sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config # 临时生效 - sudo setenforce 0 + setenforce 0 } @@ -199,11 +198,8 @@ fi if [[ $version -eq 7 ]]; then centos7 fi - -if ! test -e $dss_basepath/dss/visualis/build.zip; then -echo "Error*************:用户自行编译安装DSS WEB时，则需要把visualis的前端安装包build.zip放置于$dss_basepath/dss/visualis用于自动化安装" -exit 1 -fi - +echo '安装visualis前端,用户自行编译DSS前端安装包，则安装时需要把visualis的前端安装包放置于此'$dss_basepath/dss/visualis'，用于自动化安装:' cd $dss_basepath/dss/visualis;unzip -o build.zip > /dev/null -#echo "请浏览器访问：http://${dss_nginx_ip}:${dss_web_port}" +echo '安装linkis管理台,用户自行编译DSS前端安装包，则安装时需要把linkis管理台安装包放置于此'$dss_basepath/dss/linkis'，用于自动化安装:' +cd $dss_basepath/dss/linkis;unzip -o build.zip > /dev/null +echo "请浏览器访问：http://${dss_ipaddr}:${dss_port}" diff --git a/web/mock.js b/web/mock.js old mode 100644 new mode 100755 index b74917682..82953f9b4 --- a/web/mock.js +++ b/web/mock.js @@ -15,5 +15,5 @@ * */ -module.exports = function (app, Mock) { +module.exports = function () { } diff --git a/web/package.json b/web/package.json old mode 100644 new mode 100755 index f0f6dd869..fca84cd90 --- a/web/package.json +++ b/web/package.json @@ -1,15 +1,20 @@ { "name": "dataspherestudio", - "version": "0.9.1", + "version": "1.0.0", "private": true, "scripts": { "serve": "vue-cli-service serve", + "serve-workflows": "npm run serve --module=scriptis,workflows --micro_module=workflows", + "serve-apiServices": "npm run serve --module=apiServices,workspace --micro_module=apiServices", "build": "vue-cli-service build", "build-sandbox": "vue-cli-service build --mode sandbox", - "release": "node release.js", + "build-workflows": "npm run build --module=scriptis,workflows --micro_module=workflows", + "build-apiServices": "npm run build --module=apiServices,workspace --micro_module=apiServices", + "release": "node ./scripts/release.js", "lint": "vue-cli-service lint --no-fix", "fix": "eslint --ext .js,.vue src --fix", - "precommit": "lint-staged" + "precommit": "lint-staged", + "postinstall": "patch-package" }, "husky": { "hooks": { @@ -24,48 +29,51 @@ ] }, "dependencies": { - "axios": "^0.19.0", - "babel-polyfill": "^6.26.0", - "core-js": "^2.6.5", - "dexie": "^2.0.4", - "dt-sql-parser": "^1.1.10", - "highlight.js": "^9.12.0", - "iview": "^3.4.2", - "lodash": "^4.17.10", - "md5": "^2.2.1", - "mitt": "^1.1.3", - "moment": "^2.22.2", - "monaco-editor": "^0.20.0", - "pinyin": "^2.9.0", - "qs": "^6.9.1", - "reconnecting-websocket": "^4.1.10", - "sql-formatter": "^2.1.0", + "axios": "0.19.2", + "babel-polyfill": "6.26.0", + "core-js": "2.6.11", + "dexie": "2.0.4", + "dt-sql-parser": "1.2.1", + "eslint": "6.8.0", + "eslint-plugin-vue": "6.2.2", + "highlight.js": "9.18.3", + "iview": "3.5.4", + "lodash": "4.17.20", + "md5": "2.3.0", + "mitt": "1.2.0", + "moment": "2.29.1", + "monaco-editor": "0.19.3", + "pinyin": "2.9.1", + "qs": "6.9.4", + "reconnecting-websocket": "4.4.0", + "sql-formatter": "2.3.3", "svgo": "1.3.0", - "vue": "^2.6.6", - "vue-i18n": "^8.14.1", - "vue-router": "^3.0.1", - "vuescroll": "^4.13.1" + "vue": "2.6.12", + "vue-i18n": "8.22.1", + "vue-router": "3.4.8", + "vuedraggable": "2.24.3", + "vuescroll": "4.16.1", + "worker-loader": "2.0.0" }, "devDependencies": { - "@kazupon/vue-i18n-loader": "^0.4.1", - "@vue/cli-plugin-babel": "^3.8.0", - "@vue/cli-plugin-eslint": "^3.8.0", - "@vue/cli-service": "^3.8.0", - "@vue/eslint-config-standard": "^4.0.0", - "archiver": "^3.0.3", - "babel-eslint": "^10.0.1", - "copy-webpack-plugin": "^4.5.2", - "eslint": "^6.5.1", - "eslint-config-google": "^0.13.0", - "eslint-plugin-prettier": "^3.1.0", - "eslint-plugin-vue": "^5.0.0", - "filemanager-webpack-plugin": "^2.0.5", - "husky": "^1.2.0", - "lint-staged": "^8.1.0", - "monaco-editor-webpack-plugin": "^1.7.0", - "node-sass": "^4.9.0", - "sass-loader": "^7.1.0", - "vue-cli-plugin-mockjs": "^0.1.3", - "vue-template-compiler": "^2.6.10" + "@kazupon/vue-i18n-loader": "0.4.1", + "@vue/cli-plugin-babel": "3.12.1", + "@vue/cli-plugin-eslint": "3.12.1", + "@vue/cli-service": "3.12.1", + "@vue/eslint-config-standard": "4.0.0", + "archiver": "3.1.1", + "babel-eslint": "10.1.0", + "copy-webpack-plugin": "4.6.0", + "filemanager-webpack-plugin": "2.0.5", + "husky": "1.3.1", + "lint-staged": "8.2.1", + "monaco-editor-webpack-plugin": "1.8.2", + "node-sass": "4.14.1", + "patch-package": "6.2.2", + "sass-loader": "7.3.1", + "svg-sprite-loader": "5.0.0", + "vue-cli-plugin-mockjs": "0.1.3", + "vue-template-compiler": "2.6.12", + "webpack-virtual-modules": "0.3.2" } } diff --git a/web/patches/iview+3.5.4.patch b/web/patches/iview+3.5.4.patch new file mode 100755 index 000000000..033932c96 --- /dev/null +++ b/web/patches/iview+3.5.4.patch @@ -0,0 +1,78 @@ +diff --git a/node_modules/iview/dist/iview.js b/node_modules/iview/dist/iview.js +index 1de5eb9..be7b8dd 100644 +--- a/node_modules/iview/dist/iview.js ++++ b/node_modules/iview/dist/iview.js +@@ -6134,9 +6134,12 @@ var Popper = isServer ? function () {} : __webpack_require__(106);exports.defaul + this.$on('on-destroy-popper', this.destroy); + }, + beforeDestroy: function beforeDestroy() { ++ this.$off('on-update-popper', this.update); ++ this.$off('on-destroy-popper', this.destroy); + if (this.popper) { + this.popper.destroy(); + } ++ this.popper = null; + } + }; + +@@ -7159,6 +7162,7 @@ function destroy() { + if (this.options.removeOnDestroy) { + this.popper.parentNode.removeChild(this.popper); + } ++ this.reference = null + return this; + } + +@@ -7210,7 +7214,7 @@ function setupEventListeners(reference, options, state, updateBound) { + * @memberof Popper + */ + function enableEventListeners() { +- if (!this.state.eventsEnabled) { ++ if (this.state.eventsEnabled === true) { + this.state = setupEventListeners(this.reference, this.options, this.state, this.scheduleUpdate); + } + } +@@ -7247,7 +7251,7 @@ function removeEventListeners(reference, state) { + */ + function disableEventListeners() { + if (this.state.eventsEnabled) { +- cancelAnimationFrame(this.scheduleUpdate); ++ // cancelAnimationFrame(this.scheduleUpdate); + this.state = removeEventListeners(this.reference, this.state); + } + } +@@ -8614,9 +8618,10 @@ var Popper = function () { + var options = arguments.length > 2 && arguments[2] !== undefined ? arguments[2] : {}; + classCallCheck(this, Popper); + +- this.scheduleUpdate = function () { +- return requestAnimationFrame(_this.update); +- }; ++ // this.scheduleUpdate = function () { ++ // return requestAnimationFrame(_this.update); ++ // }; ++ this.scheduleUpdate = _this.update.bind(this) + + // make update() debounced, so that it only runs at most once-per-tick + this.update = debounce(this.update.bind(this)); +@@ -8664,7 +8669,6 @@ var Popper = function () { + + // fire the first update to position the popper in the right place + this.update(); +- + var eventsEnabled = this.options.eventsEnabled; + if (eventsEnabled) { + // setup event listeners, they will take care of update the position in specific situations +diff --git a/node_modules/iview/src/components/select/dropdown.vue b/node_modules/iview/src/components/select/dropdown.vue +index c36bb28..70d98c8 100644 +--- a/node_modules/iview/src/components/select/dropdown.vue ++++ b/node_modules/iview/src/components/select/dropdown.vue +@@ -67,7 +67,7 @@ + }, + onUpdate:()=>{ + this.resetTransformOrigin(); +- } ++ }, + }); + }); + } \ No newline at end of file diff --git a/web/postcss.config.js b/web/postcss.config.js old mode 100644 new mode 100755 diff --git a/web/public/favicon.ico b/web/public/favicon.ico old mode 100644 new mode 100755 diff --git a/web/public/index.html b/web/public/index.html old mode 100644 new mode 100755 diff --git a/web/find.js b/web/scripts/find.js old mode 100644 new mode 100755 similarity index 98% rename from web/find.js rename to web/scripts/find.js index cc9cb71d8..d71f83ee4 --- a/web/find.js +++ b/web/scripts/find.js @@ -61,7 +61,6 @@ function findCNText(filePath) { if (err) { throw err; } - console.log('done.'); }) } if (isDir) { diff --git a/web/release.js b/web/scripts/release.js old mode 100644 new mode 100755 similarity index 100% rename from web/release.js rename to web/scripts/release.js diff --git a/web/replace.js b/web/scripts/replace.js old mode 100644 new mode 100755 similarity index 97% rename from web/replace.js rename to web/scripts/replace.js index a34a18d74..b9a1f214d --- a/web/replace.js +++ b/web/scripts/replace.js @@ -34,9 +34,7 @@ function replacefnT(filePath) { var isFile = stats.isFile(); var isDir = stats.isDirectory(); if (isFile) { - // console.log(filedir); var content = fs.readFileSync(filedir, 'utf-8'); - var values = Object.values(json.translate); var keys = Object.keys(json.translate) values = values.sort((a, b) => b.length - a.length); @@ -49,7 +47,6 @@ function replacefnT(filePath) { // placeholder="dd" // label="dd" // 未通过校验 - // js // '技术规则' var attrReg = new RegExp(` (title|name|label|placeholder)="${it}"`, 'g') @@ -66,7 +63,6 @@ function replacefnT(filePath) { throw err; } // eslint-disable-next-line no-console - console.log('done.'); }) } if (isDir) { diff --git a/web/translate.js b/web/scripts/translate.js old mode 100644 new mode 100755 similarity index 100% rename from web/translate.js rename to web/scripts/translate.js diff --git a/web/src/apps/apiServices/assets/images/dssLogo5.png b/web/src/apps/apiServices/assets/images/dssLogo5.png new file mode 100755 index 000000000..a16cac58b Binary files /dev/null and b/web/src/apps/apiServices/assets/images/dssLogo5.png differ diff --git a/web/src/apps/apiServices/assets/images/dssLogo5_1.png b/web/src/apps/apiServices/assets/images/dssLogo5_1.png new file mode 100755 index 000000000..e4d9b1619 Binary files /dev/null and b/web/src/apps/apiServices/assets/images/dssLogo5_1.png differ diff --git a/web/src/apps/apiServices/i18n/en.json b/web/src/apps/apiServices/i18n/en.json new file mode 100755 index 000000000..05923fe8c --- /dev/null +++ b/web/src/apps/apiServices/i18n/en.json @@ -0,0 +1,262 @@ +{ + "message": { + "apiServices": { + "viewAll": "View All", + "viewValid": "View Valid", + "deleted": "Deleted", + "delete": "Delete", + "enterDetele": "confirm Detele", + "enterDeteleName": "Are you sure to delete {name}?", + "publishing": "Publishing", + "confirm": "Confirm", + "modifyRemarks": "Modify Remarks", + "save": "Save", + "open": "Open", + "title": "Data Service", + "back": "Back", + "tableDisplay": "Table", + "cardDisplay": "Card", + "display": "Display", + "action": "Aciton", + "kill": { + "title": "Execution Notice", + "desc": "Stopped executing this script" + }, + "date": { + "label": " Start Date", + "placeholder": "Please choose the start date" + }, + "query": { + "apiName": "Data Service Title", + "tag": "Tag", + "status": "Service Status", + "commiter": "Commiter", + "buttonText": "Query", + "clearButtonText": "Clear", + "stop": "Stop", + "comment": "Comment", + "more": "More" + }, + "formTitle": { + "dataApiManage": "Data API Manage" + }, + "shortcuts": { + "week": "Recent Week", + "month": "Recent Month", + "threeMonths": "Recent Three Months" + }, + "placeholder": { + "enterName": "Please enter the name", + "apiName": "Enter API Name", + "tag": "Select Tag", + "scope": "Select Scope", + "description": "Enter Description", + "visibleRole": "Select Visible Role", + "emter": "Enter", + "startDate": "Please choose the start date", + "dateAndTime": "Select Date And Time", + "inputSubmitter": "Input Submitter" + }, + "label": { + "apiName": "API Name：", + "tag": "Tag：", + "name": "Name", + "parameter": "Parameter Info：", + "scope": "Scope：", + "description": "Description：", + "visibleRole": "Visible Role：", + "test": "Test：", + "applicant": "Applicant：", + "status": "Status：", + "find": "Find", + "path": "API Path：", + "submitter": "Submitter", + "scriptsPath": "Scripts Path：" + }, + "buttonText": { + "enter": "Enter", + "maintained": "Being maintained", + "refresh": "Refresh Status" + }, + "tip": { + "success": "Success", + "cannotBeEmpty": "Cannot Be Empty", + "failed": "Failed", + "disableConfirmTitle": "Disable Confirm", + "disableConfirm": "Are you sure you want to disable the {name} API ?", + "enableConfirmTitle": "Enable Confirm", + "enableConfirm": "Are you sure you want to enable the {name} API ?" + }, + "apiTable": { + "user": "User", + "duration": "Duration", + "reason": "Reason", + "ip_whitelist": "Ip Whitelist", + "apply_time": "Apply Time", + "caller": "Caller", + "access": "Access", + "version": "Version", + "scriptPath": "Script Path", + "creator": "Creator", + "publishDateStr": "Publish DateStr", + "updateDateStr": "Update Date", + "apiName": "Data Service Name", + "apiPath": "API Path", + "status": "Status", + "type": "Type", + "describe": "Describe", + "calledCount": "Call count", + "responsiblePerson": "Responsible Person", + "belongTo": "Belong To", + "tag": "Tag", + "operation": { + "title": "Operation", + "test": "Test", + "copy": "Copy", + "disable": "Disable", + "enable": "Enable", + "manager": "Manager", + "update": "Update", + "view": "View" + }, + "reconfirm": { + "title": "Confirm", + "disable": "Confirm disable this API？", + "enable": "Confirm enable this API？" + } + }, + "view": { + "scriptPath": "Script Path:", + "scriptContent": "Script Content:" + }, + "enable": "Running", + "disable": "Stop", + "normal": "Normal", + "expired": "Expired", + "all": "All", + "updateApiModal": { + "modalTitle": "Update API", + "selectApi": "API Name", + "apiPath": "API Path", + "apiVersion": "API Version", + "apiVersionUpgrade": "New Version", + "paramConfirm": "Please confirm param" + }, + "paramTable": { + "paramName": "Param Name", + "paramType": "Param Type", + "require": { + "title": "Require", + "yes": "Yes", + "no": "No", + "hide": "Hide" + }, + "defaultValue": "Default Value", + "displayName": "Display Name", + "describe": "Describe" + }, + "rule": { + "nameRule": "Please input api name!", + "pathRule": "Please input api path!", + "pathRegRule": "Api path illegal!", + "requestTypeRule": "Please select request type!", + "protocolRule": "Please select protocol!", + "contentLengLimit": "Length limit 255", + "pathRepeat": "Api path exists", + "nameRepeat": "Api name exists" + }, + "visible": { + "workspace": "Workspace", + "personal": "Personal", + "public": "Public" + }, + "apiInfo": { + "title": "API Info", + "authorize": "Token Authorize", + "set": "Set", + "tag": "Tag", + "metrics": "API Metrics", + "monthCall": "Call Total This Month", + "weekCall": "Call Total This Week", + "dayCall": "Call Total Today", + "averageTime": "Average Call Time", + "callSuccess": "Success Percent", + "topQps": "TOP QPS", + "use": "API Call Info", + "selectCallTime": "Select Call Time", + "version": "Version", + "callStatus": "Call Info", + "callUser": "Caller", + "callApp": "CallApp", + "callIp": "Call IP", + "callTime": "Call Time", + "callVersion": "Call Version", + "operation": "Operation", + "query": { + "clearButtonText": "Clear", + "buttonText": "Query" + } + }, + "apiSetForm": { + "flowLimitation": "Flow Limitation", + "defaultSetting": "Default Setting", + "advancedSetting": "Advanced Setting", + "form": { + "MaxConcurrentNumber": "Max Concurrent Number：", + "MaxSimultaneousQuery": "Max Simultaneous Query：", + "MaxTimeQuery": "Max Time Query：", + "defaultVersion": "Default Version：", + "defaultConfigurationParameter": "Default Configuration Parameter：", + "defaultVariable": "Default Variable：", + "approvalNo": "Approval No：" + } + }, + "apiTestInfo": { + "title": "API Test", + "testApi": "Test API", + "apiVersion": "Version", + "search": "Search", + "params": "Screening criteria", + "moreParams": "More filters", + "paramsMore": "More filters", + "selectParams": "已选择过滤条件", + "allSelect": "全选", + "allSelectParams": "可选择参数列表", + "paramTable": { + "paramName": "Param Name", + "paramType": "Param Type", + "description": "Param Description", + "require": { + "title": "Require", + "yes": "YES", + "no": "NO" + }, + "defaultValue": "Test Value" + }, + "testResult": "Test Result" + }, + "apiVersionInfo": { + "title": "API Version", + "params": "params", + "versionInfoTable": { + "version": "Version", + "status": "Status", + "source": "Source", + "creator": "Publisher", + "publishDate": "Publish Time", + "updateDate": "Update Time", + "operation": "Operation", + "query": { + "title": "Query", + "clear": "Clear", + "publishDate": "Select Publish Time" + } + } + }, + "notice": { + "publishSuccess": "Publish Success" + }, + "tagPlaceholder": "tag，click enter add" + } + } +} diff --git a/web/src/apps/apiServices/i18n/zh.json b/web/src/apps/apiServices/i18n/zh.json new file mode 100755 index 000000000..cd6505c21 --- /dev/null +++ b/web/src/apps/apiServices/i18n/zh.json @@ -0,0 +1,262 @@ +{ + "message": { + "apiServices": { + "viewAll": "显示全部", + "viewValid": "屏蔽已删除", + "deleted": "已删除", + "delete": "删除", + "enterDetele": "确认删除", + "enterDeteleName": "确认删除{name}服务?", + "publishing": "发布中", + "confirm": "确认", + "modifyRemarks": "修改备注", + "save": "保存", + "open": "打开", + "title": "数据服务", + "back": "返回", + "tableDisplay": "列表展示", + "cardDisplay": "图标展示", + "display": "展示方式", + "action": "操作", + "kill": { + "title": "运行提示", + "desc": "已经停止执行脚本" + }, + "date": { + "label": "起始时间", + "placeholder": "请选择起始日期" + }, + "query": { + "apiName": "数据服务名称", + "tag": "标签", + "status": "服务状态", + "commiter": "提交人", + "buttonText": "查询", + "clearButtonText": "清空", + "stop": "停止", + "comment": "备注", + "more": "更多" + }, + "formTitle": { + "dataApiManage": "数据API管理" + }, + "shortcuts": { + "week": "最近一周", + "month": "最近一个月", + "threeMonths": "最近三个月" + }, + "placeholder": { + "enterName": "请输入名称", + "apiName": "请输入API名称", + "tag": "请选择标签", + "scope": "请选择可见范围", + "description": "请输入描述", + "visibleRole": "请选择可见角色", + "emter": "请输入", + "startDate": "请选择起始日期", + "dateAndTime": "请选择日期和时间", + "inputSubmitter": "输入提交人" + }, + "label": { + "apiName": "API名称：", + "tag": "标签：", + "name": "名称", + "parameter": "参数信息：", + "scope": "可见范围：", + "description": "描述：", + "visibleRole": "可见角色：", + "test": "测试：", + "applicant": "申请人：", + "status": "状态：", + "path": "API路径：", + "submitter": "提交人", + "find": "查询", + "scriptsPath": "脚本路径：" + }, + "buttonText": { + "enter": "进入使用", + "maintained": "禁止使用", + "refresh": "刷新状态" + }, + "tip": { + "success": "保存成功", + "failed": "不通过", + "disableConfirmTitle": "禁用确认", + "enableConfirmTitle": "开启确认", + "disableConfirm": "确认禁用 {name} api吗？", + "enableConfirm": "确认开启 {name} api吗？", + "cannotBeEmpty": "不能为空！" + }, + "apiTable": { + "apiName": "数据服务名称", + "apiPath": "API 路径", + "status": "状态", + "type": "类型", + "describe": "描述", + "calledCount": "调用次数", + "responsiblePerson": "责任人", + "belongTo": "API归属", + "tag": "标签", + "user": "申请用户", + "duration": "生效时长", + "reason": "申请原因", + "ip_whitelist": "白名单IP", + "apply_time": "申请起始时间", + "caller": "调用方", + "access": "限流情况", + "version": "版本号", + "scriptPath": "来源", + "creator": "发布者", + "publishDateStr": "发布时间", + "updateDateStr": "更新时间", + "operation": { + "title": "操作", + "test": "测试", + "copy": "复制", + "disable": "禁用", + "enable": "启用", + "manager": "管理", + "update": "更新", + "view": "查看" + }, + "reconfirm": { + "title": "确认", + "disable": "确定禁用该API吗？", + "enable": "确定启动该API吗？" + } + }, + "view": { + "scriptPath": "脚本路径：", + "scriptContent": "脚本内容：" + }, + "enable": "运行中", + "disable": "已禁用", + "normal": "正常", + "expired": "已过期", + "all": "全部", + "updateApiModal": { + "modalTitle": "数据服务API更新", + "selectApi": "API 名称", + "apiPath": "API 路径", + "apiVersion": "API版本", + "apiVersionUpgrade": "生成新版本", + "paramConfirm": "请确认参数" + }, + "paramTable": { + "paramName": "参数名称", + "paramType": "参数类型", + "require": { + "title": "是否必填", + "yes": "是", + "no": "否", + "hide": "隐藏" + }, + "defaultValue": "默认值", + "displayName": "展示名", + "describe": "描述" + }, + "rule": { + "nameRule": "请填写API名称", + "pathRule": "请填写API路径", + "pathRegRule": "API路径不合法", + "requestTypeRule": "请选择请求方式", + "protocolRule": "请选择协议", + "contentLengthLimit": "长度不能大于255", + "pathRepeat": "API路径已存在", + "nameRepeat": "API名称已存在" + }, + "visible": { + "workspace": "工作空间", + "personal": "个人", + "public": "公开" + }, + "notice": { + "publishSuccess": "发布成功" + }, + "apiVersionInfo": { + "title": "API版本", + "versionInfoTable": { + "version": "版本", + "status": "状态", + "source": "来源", + "creator": "发布者", + "publishDate": "发布时间", + "updateDate": "最后更新时间", + "operation": "操作", + "query": { + "title": "查询", + "clear": "清空", + "publishDate": "请选择发布时间" + } + } + }, + "apiTestInfo": { + "title": "API测试", + "testApi": "测试API", + "apiVersion": "请求版本", + "search": "搜索", + "params": "筛选条件", + "moreParams": "更多筛选条件...", + "paramsMore": "筛选更多", + "selectParams": "已选择过滤条件", + "allSelect": "全选", + "allSelectParams": "可选择参数列表", + "paramTable": { + "paramName": "参数名称", + "paramType": "参数类型", + "description": "参数描述", + "displayName": "展示名", + "require": { + "title": "是否必填", + "yes": "是", + "no": "否" + }, + "defaultValue": "测试值" + }, + "testResult": "测试结果" + }, + "apiSetForm": { + "flowLimitation": "限流设置", + "defaultSetting": "默认设置", + "advancedSetting": "高级设置", + "form": { + "MaxConcurrentNumber": "最大并发数：", + "MaxSimultaneousQuery": "最大同时在查数：", + "MaxTimeQuery": "最大查询时长：", + "defaultVersion": "默认版本：", + "defaultConfigurationParameter": "默认配置参数：", + "defaultVariable": "默认变量：", + "approvalNo": "审批单号：" + } + }, + "apiInfo": { + "title": "API信息", + "set": "设置", + "authorize": "Token授权", + "tag": "标签", + "metrics": "API Metrics", + "monthCall": "本月总调用次数", + "weekCall": "本周总调用次数", + "dayCall": "今日调用次数", + "averageTime": "平均调用时长", + "callSuccess": "调用成功率", + "topQps": "最高QPS", + "use": "API调用情况", + "selectCallTime": "请选择调用时间", + "version": "版本", + "callStatus": "调用情况", + "callUser": "调用者", + "callApp": "调用方", + "callIp": "调用IP", + "callTime": "调用时长", + "callVersion": "调用版本", + "operation": "操作", + "query": { + "clearButtonText": "清空", + "buttonText": "查询" + } + }, + "tagPlaceholder": "标签，按 enter 创建" + } + } +} diff --git a/web/src/apps/apiServices/module/apiServices/apiCard.vue b/web/src/apps/apiServices/module/apiServices/apiCard.vue new file mode 100755 index 000000000..a3442b138 --- /dev/null +++ b/web/src/apps/apiServices/module/apiServices/apiCard.vue @@ -0,0 +1,190 @@ + + + + diff --git a/web/src/apps/apiServices/module/apiServices/dropdownCom.vue b/web/src/apps/apiServices/module/apiServices/dropdownCom.vue new file mode 100755 index 000000000..e75d4725f --- /dev/null +++ b/web/src/apps/apiServices/module/apiServices/dropdownCom.vue @@ -0,0 +1,35 @@ + + + diff --git a/web/src/apps/apiServices/module/apiServices/form.vue b/web/src/apps/apiServices/module/apiServices/form.vue new file mode 100755 index 000000000..6d0423b21 --- /dev/null +++ b/web/src/apps/apiServices/module/apiServices/form.vue @@ -0,0 +1,181 @@ +v_wbqxie(谢乾) 8-14 16:57:38 + + + diff --git a/web/src/apps/apiServices/module/apiServices/index.js b/web/src/apps/apiServices/module/apiServices/index.js new file mode 100755 index 000000000..72d7cc4c8 --- /dev/null +++ b/web/src/apps/apiServices/module/apiServices/index.js @@ -0,0 +1,10 @@ +export default { + name: 'apiServices', + events: [], + dispatchs: { + }, + data: { + API_PATH: process.env.VUE_APP_MN_CONFIG_PREFIX || `http://${window.location.host}/api/rest_j/v1/`, + }, + component: () => import('./index.vue'), +}; \ No newline at end of file diff --git a/web/src/apps/apiServices/module/apiServices/index.vue b/web/src/apps/apiServices/module/apiServices/index.vue new file mode 100755 index 000000000..491ea5232 --- /dev/null +++ b/web/src/apps/apiServices/module/apiServices/index.vue @@ -0,0 +1,750 @@ + + + + diff --git a/web/src/apps/apiServices/module/apiServicesExecute/index.js b/web/src/apps/apiServices/module/apiServicesExecute/index.js new file mode 100755 index 000000000..45967acc7 --- /dev/null +++ b/web/src/apps/apiServices/module/apiServicesExecute/index.js @@ -0,0 +1,11 @@ +export default { + name: 'dataServicesMangement', + events: [], + dispatchs: { + IndexedDB: ['updateResult'], + }, + data: { + API_PATH: process.env.VUE_APP_MN_CONFIG_PREFIX || `http://${window.location.host}/api/rest_j/v1/`, + }, + component: () => import('./index.vue'), +}; \ No newline at end of file diff --git a/web/src/apps/apiServices/module/apiServicesExecute/index.vue b/web/src/apps/apiServices/module/apiServicesExecute/index.vue new file mode 100755 index 000000000..6fd78d8ef --- /dev/null +++ b/web/src/apps/apiServices/module/apiServicesExecute/index.vue @@ -0,0 +1,351 @@ + + + + diff --git a/web/src/apps/apiServices/module/header/index.js b/web/src/apps/apiServices/module/header/index.js new file mode 100755 index 000000000..4e0a78d62 --- /dev/null +++ b/web/src/apps/apiServices/module/header/index.js @@ -0,0 +1,5 @@ +export default { + name: 'scriptisHeader', + component: () => import('./index.vue'), + dispatchs: [], +}; \ No newline at end of file diff --git a/web/src/apps/apiServices/module/header/index.scss b/web/src/apps/apiServices/module/header/index.scss new file mode 100755 index 000000000..e4ee1f7dc --- /dev/null +++ b/web/src/apps/apiServices/module/header/index.scss @@ -0,0 +1,21 @@ +/*! + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +@import '@/common/style/variables.scss'; + +@import '@/common/style/headerCommon.scss'; + diff --git a/web/src/apps/apiServices/module/header/index.vue b/web/src/apps/apiServices/module/header/index.vue new file mode 100755 index 000000000..c85f6ec48 --- /dev/null +++ b/web/src/apps/apiServices/module/header/index.vue @@ -0,0 +1,114 @@ + + + diff --git a/web/src/apps/apiServices/module/header/userMenu.vue b/web/src/apps/apiServices/module/header/userMenu.vue new file mode 100755 index 000000000..5efef9bdf --- /dev/null +++ b/web/src/apps/apiServices/module/header/userMenu.vue @@ -0,0 +1,120 @@ + + + \ No newline at end of file diff --git a/web/src/apps/apiServices/module/servicesMangement/component/APIInfo.vue b/web/src/apps/apiServices/module/servicesMangement/component/APIInfo.vue new file mode 100755 index 000000000..3b7554d08 --- /dev/null +++ b/web/src/apps/apiServices/module/servicesMangement/component/APIInfo.vue @@ -0,0 +1,57 @@ + + + + diff --git a/web/src/apps/apiServices/module/servicesMangement/component/APIVersions.vue b/web/src/apps/apiServices/module/servicesMangement/component/APIVersions.vue new file mode 100755 index 000000000..1035cab4b --- /dev/null +++ b/web/src/apps/apiServices/module/servicesMangement/component/APIVersions.vue @@ -0,0 +1,281 @@ + + + + diff --git a/web/src/apps/apiServices/module/servicesMangement/component/TokenAuthorization.vue b/web/src/apps/apiServices/module/servicesMangement/component/TokenAuthorization.vue new file mode 100755 index 000000000..789f1db13 --- /dev/null +++ b/web/src/apps/apiServices/module/servicesMangement/component/TokenAuthorization.vue @@ -0,0 +1,217 @@ + + + + diff --git a/web/src/apps/apiServices/module/servicesMangement/index.js b/web/src/apps/apiServices/module/servicesMangement/index.js new file mode 100755 index 000000000..868454055 --- /dev/null +++ b/web/src/apps/apiServices/module/servicesMangement/index.js @@ -0,0 +1,10 @@ +export default { + name: 'servicesMangement', + events: [], + dispatchs: { + }, + data: { + API_PATH: process.env.VUE_APP_MN_CONFIG_PREFIX || `http://${window.location.host}/api/rest_j/v1/`, + }, + component: () => import('./index.vue'), +}; \ No newline at end of file diff --git a/web/src/apps/apiServices/module/servicesMangement/index.vue b/web/src/apps/apiServices/module/servicesMangement/index.vue new file mode 100755 index 000000000..f007249a6 --- /dev/null +++ b/web/src/apps/apiServices/module/servicesMangement/index.vue @@ -0,0 +1,377 @@ + + + + + diff --git a/web/src/apps/apiServices/router.js b/web/src/apps/apiServices/router.js new file mode 100755 index 000000000..0d4861fa7 --- /dev/null +++ b/web/src/apps/apiServices/router.js @@ -0,0 +1,31 @@ +const routes = [ + { + path: '/apiservices', + name: 'Apiservices', + meta: { + title: 'DataSphere Studio', + publicPage: true + }, + component: () => import('./view/apiServices/index.vue') + }, + { + path: '/servicesMangement', + name: 'ServicesMangement', + meta: { + title: 'DataSphere Studio', + publicPage: true + }, + component: () => import('./view/servicesMangement/index.vue') + }, + { + path: '/servicesExecute', + name: 'ServicesExecute', + meta: { + title: 'DataSphere Studio', + publicPage: true + }, + component: () => import('./view/servicesExecute/index.vue') + }, +] + +export default routes; \ No newline at end of file diff --git a/web/src/apps/apiServices/view/apiServices/index.vue b/web/src/apps/apiServices/view/apiServices/index.vue new file mode 100755 index 000000000..3f90d2ad2 --- /dev/null +++ b/web/src/apps/apiServices/view/apiServices/index.vue @@ -0,0 +1,14 @@ + + + \ No newline at end of file diff --git a/web/src/apps/apiServices/view/servicesExecute/index.vue b/web/src/apps/apiServices/view/servicesExecute/index.vue new file mode 100755 index 000000000..b55324d67 --- /dev/null +++ b/web/src/apps/apiServices/view/servicesExecute/index.vue @@ -0,0 +1,12 @@ + + + \ No newline at end of file diff --git a/web/src/apps/apiServices/view/servicesMangement/index.vue b/web/src/apps/apiServices/view/servicesMangement/index.vue new file mode 100755 index 000000000..717da2223 --- /dev/null +++ b/web/src/apps/apiServices/view/servicesMangement/index.vue @@ -0,0 +1,15 @@ + + + \ No newline at end of file diff --git a/web/src/apps/scriptis/assets/images/dssLogo5.png b/web/src/apps/scriptis/assets/images/dssLogo5.png new file mode 100755 index 000000000..a16cac58b Binary files /dev/null and b/web/src/apps/scriptis/assets/images/dssLogo5.png differ diff --git a/web/src/apps/scriptis/assets/images/flink.svg b/web/src/apps/scriptis/assets/images/flink.svg new file mode 100755 index 000000000..9e067cfc9 --- /dev/null +++ b/web/src/apps/scriptis/assets/images/flink.svg @@ -0,0 +1,8560 @@ + + + + diff --git a/web/src/assets/styles/home.scss b/web/src/apps/scriptis/assets/styles/home.scss old mode 100644 new mode 100755 similarity index 81% rename from web/src/assets/styles/home.scss rename to web/src/apps/scriptis/assets/styles/home.scss index 7c90d1fa8..491e471d4 --- a/web/src/assets/styles/home.scss +++ b/web/src/apps/scriptis/assets/styles/home.scss @@ -15,7 +15,7 @@ * */ -@import 'variables.scss'; +@import '@/common/style/variables.scss'; .home-page{ position: $relative; @@ -37,8 +37,8 @@ .nav-list{ width: 44px; height: 100%; - border-right: $border-width-base $border-style-base #dcdee2; - background: #f8f8f9; + border-right: $border-width-base $border-style-base $border-color-base; + background: $table-thead-bg; .nav-warp{ position: $absolute; height: 100%; @@ -54,23 +54,23 @@ &.actived{ width: 44px; border-left: 3px solid $primary-color; - border-bottom: $border-width-base $border-style-base #dcdee2; - border-top: $border-width-base $border-style-base #dcdee2; - background: #fff; + border-bottom: $border-width-base $border-style-base $border-color-base; + border-top: $border-width-base $border-style-base $border-color-base; + background: $background-color-white ; color: $primary-color; } &:hover{ color: $primary-color; } .nav-icon{ - font-size: 16px; + font-size: $font-size-large; } .nav-name{ width: 100%; white-space: normal; word-break: break-all; display: inline-block; - line-height: 16px; + line-height: $font-size-large; } } } @@ -87,6 +87,14 @@ } } } + .fi-flink-sql { + background-image: url('../images/flink.svg'); + width: 20px; + background-size: contain; + background-repeat: no-repeat; + background-position: center; + margin-right: -8px; + } // @media screen and(max-height: 790px) { // .nav-list { // width: 30px; diff --git a/web/src/assets/styles/sidebar.scss b/web/src/apps/scriptis/assets/styles/sidebar.scss old mode 100644 new mode 100755 similarity index 96% rename from web/src/assets/styles/sidebar.scss rename to web/src/apps/scriptis/assets/styles/sidebar.scss index 7efed09a7..66076512d --- a/web/src/assets/styles/sidebar.scss +++ b/web/src/apps/scriptis/assets/styles/sidebar.scss @@ -15,7 +15,7 @@ * */ -@import 'variables.scss'; +@import '@/common/style/variables.scss'; .we-side-bar { height: 100%; overflow: hidden; diff --git a/web/src/apps/scriptis/components/apiPublish/TagsInput/index.scss b/web/src/apps/scriptis/components/apiPublish/TagsInput/index.scss new file mode 100755 index 000000000..e2682eec2 --- /dev/null +++ b/web/src/apps/scriptis/components/apiPublish/TagsInput/index.scss @@ -0,0 +1,39 @@ +@import '@/common/style/variables.scss'; +.tags-wrap{ + width: 100%; + height: 100%; + outline: none; + &::after{ + content: ""; + display: block; + height: 0; + clear: both; + } +} +.tags, .tags-input{ + position: relative; + float: left; + color: #fff; + line-height: 28px; + margin: 0 4px 4px 0; + padding: 0 22px 0 10px; + border-radius: $border-radius-base; + .content{ + line-height: 28px; + } + .del{ + width: 22px; + height: 28px; + text-align: center; + cursor: pointer; + position: absolute; + top: -1px; + right: 0; + } +} +.tags-input{ + background-color: inherit; + border: $border-width-base $border-style-base $border-color-base; + color: inherit; + width: auto !important; +} diff --git a/web/src/apps/scriptis/components/apiPublish/TagsInput/index.vue b/web/src/apps/scriptis/components/apiPublish/TagsInput/index.vue new file mode 100755 index 000000000..9d28fb239 --- /dev/null +++ b/web/src/apps/scriptis/components/apiPublish/TagsInput/index.vue @@ -0,0 +1,85 @@ + + + + + diff --git a/web/src/js/component/functionSetting/index.js b/web/src/apps/scriptis/components/functionSetting/index.js old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/functionSetting/index.js rename to web/src/apps/scriptis/components/functionSetting/index.js diff --git a/web/src/apps/scriptis/components/functionSetting/index.scss b/web/src/apps/scriptis/components/functionSetting/index.scss new file mode 100755 index 000000000..b14cbe634 --- /dev/null +++ b/web/src/apps/scriptis/components/functionSetting/index.scss @@ -0,0 +1,58 @@ +/*! + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +@import "@/common/style/variables.scss"; +.fn-setting { + .setting-content { + padding-right: 16px; + overflow-y: auto; + .preview { + overflow: hidden; + text-overflow: ellipsis; + width: 380px; + white-space: nowrap; + + .preview-name { + color: $success-color; + } + + .preview-params { + color: $primary-color; + } + + .preview-type { + color: $warning-color; + } + } + .format-div { + display: flex; + .format-item { + width: 50%; + } + .format-input { + padding-right: 6px; + } + } + } + .fn-default-load { + position: absolute; + left: 30px; + bottom: 20px; + } + .fn-sure-btn { + margin-right: 10px; + } +} diff --git a/web/src/apps/scriptis/components/functionSetting/index.vue b/web/src/apps/scriptis/components/functionSetting/index.vue new file mode 100755 index 000000000..49bb28c0c --- /dev/null +++ b/web/src/apps/scriptis/components/functionSetting/index.vue @@ -0,0 +1,644 @@ + + + diff --git a/web/src/js/component/functionSetting/jarPreview.vue b/web/src/apps/scriptis/components/functionSetting/jarPreview.vue old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/functionSetting/jarPreview.vue rename to web/src/apps/scriptis/components/functionSetting/jarPreview.vue diff --git a/web/src/js/component/functionSetting/pyPreview.vue b/web/src/apps/scriptis/components/functionSetting/pyPreview.vue old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/functionSetting/pyPreview.vue rename to web/src/apps/scriptis/components/functionSetting/pyPreview.vue diff --git a/web/src/js/component/functionSetting/scalaPreview.vue b/web/src/apps/scriptis/components/functionSetting/scalaPreview.vue old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/functionSetting/scalaPreview.vue rename to web/src/apps/scriptis/components/functionSetting/scalaPreview.vue diff --git a/web/src/js/component/functionSetting/usePreview.vue b/web/src/apps/scriptis/components/functionSetting/usePreview.vue old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/functionSetting/usePreview.vue rename to web/src/apps/scriptis/components/functionSetting/usePreview.vue diff --git a/web/src/js/component/functionShare/index.js b/web/src/apps/scriptis/components/functionShare/index.js old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/functionShare/index.js rename to web/src/apps/scriptis/components/functionShare/index.js diff --git a/web/src/apps/scriptis/components/functionShare/index.vue b/web/src/apps/scriptis/components/functionShare/index.vue new file mode 100755 index 000000000..e15819311 --- /dev/null +++ b/web/src/apps/scriptis/components/functionShare/index.vue @@ -0,0 +1,161 @@ + + diff --git a/web/src/apps/scriptis/components/importToHive/fieldList.vue b/web/src/apps/scriptis/components/importToHive/fieldList.vue new file mode 100755 index 000000000..4fc15ebeb --- /dev/null +++ b/web/src/apps/scriptis/components/importToHive/fieldList.vue @@ -0,0 +1,133 @@ + + diff --git a/web/src/js/component/importToHive/index.js b/web/src/apps/scriptis/components/importToHive/index.js old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/importToHive/index.js rename to web/src/apps/scriptis/components/importToHive/index.js diff --git a/web/src/apps/scriptis/components/importToHive/index.scss b/web/src/apps/scriptis/components/importToHive/index.scss new file mode 100755 index 000000000..66b756a2d --- /dev/null +++ b/web/src/apps/scriptis/components/importToHive/index.scss @@ -0,0 +1,141 @@ +/*! + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +@import '~@/common/style/variables.scss'; +.we-import-to-hive { + min-height: 480px; + padding: 20px 20px 0; + padding-top: 0; + color: #787878; + position: relative; + overflow: hidden; + &.first-step { + overflow: auto; + } + .we-import-to-hive-steps { + padding-bottom: 9px; + } + .we-import-to-hive-form { + .step-progress { + height: 500px; + position: absolute; + width: 740px; + display: flex; + z-index: 999; + background-color: rgba(0, 0, 0, 0.3); + align-items: center; + justify-content: center; + } + .step-form-title { + font-size: $font-size-large; + color: $primary-color ; + display: block; + margin: 0 0 9px; + } + .step-form-field { + display: inline-block; + margin-right: 10px; + } + .step-form-extras { + margin: 6px 4px 0 5px; + } + .seconed-Form { + display: flex; + flex-direction: column; + .field-list-wrap { + .field-list-item { + .field-list-item-child { + margin-bottom: 20px; + display: inline-block; + .field-list-item-child-label { + display: inline-block; + width: 30px; + text-align: right; + vertical-align: middle; + margin-right: 10px; + margin-left: 4px; + } + .field-list-item-child-type, + .field-list-item-child-comment, + .field-list-item-child-option, + .field-list-item-child-length, + .field-list-item-child-time { + display: inline-block; + margin: 0 2px; + } + .field-list-item-child-type { + width: 104px; + } + .field-list-item-child-option { + width: 81px; + } + .field-list-item-child-length { + width: 166px; + } + .field-list-item-child-time { + width: 122px; + } + .field-list-item-child-comment { + width: 104px; + } + } + .step-form-close, + .step-form-extras { + cursor: pointer; + } + .step-form-close { + padding-top: 6px; + } + } + } + } + .step-form-type-supplement { + margin-left: 157px; + } + .item-auto-complete { + .ivu-select-dropdown-list { + height: 300px; + } + } + .form-item-help-icon { + position: absolute; + left: -12px; + top: 10px; + } + .step-form-fields { + display: inline-flex; + .form-item-inline { + width: 220px; + display: inline-block; + } + } + } +} + +.we-import-to-hive-errmsg { + color: $error-color; + position: absolute; + left: 24px; + font-size: 12px; + display: inline-block; + width: 380px; + text-align: left; +} + +.ivu-auto-complete.ivu-select-dropdown { + max-height: 120px; +} diff --git a/web/src/apps/scriptis/components/importToHive/index.vue b/web/src/apps/scriptis/components/importToHive/index.vue new file mode 100755 index 000000000..2d829c173 --- /dev/null +++ b/web/src/apps/scriptis/components/importToHive/index.vue @@ -0,0 +1,836 @@ + + + diff --git a/web/src/js/component/newDialog/index.js b/web/src/apps/scriptis/components/newDialog/index.js old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/newDialog/index.js rename to web/src/apps/scriptis/components/newDialog/index.js diff --git a/web/src/apps/scriptis/components/newDialog/index.vue b/web/src/apps/scriptis/components/newDialog/index.vue new file mode 100755 index 000000000..a952acc5a --- /dev/null +++ b/web/src/apps/scriptis/components/newDialog/index.vue @@ -0,0 +1,260 @@ + + diff --git a/web/src/js/component/uploadDialog/index.js b/web/src/apps/scriptis/components/uploadDialog/index.js old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/uploadDialog/index.js rename to web/src/apps/scriptis/components/uploadDialog/index.js diff --git a/web/src/apps/scriptis/components/uploadDialog/index.scss b/web/src/apps/scriptis/components/uploadDialog/index.scss new file mode 100755 index 000000000..6b485785a --- /dev/null +++ b/web/src/apps/scriptis/components/uploadDialog/index.scss @@ -0,0 +1,85 @@ +/*! + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +@import '@/common/style/variables.scss'; + +.we-upload { + .we-upload-content { + width: 310px; + min-height: 230px; + padding-top: 15px; + padding-bottom: 15px; + cursor: default; + + .el-upload-dragger { + height: 220px; + } + + .el-upload-list { + max-height: 300px; + overflow: auto; + } + + .we-upload_path { + color: #97a8be; + font-size: 12px; + padding-top: 40px; + } + } + + .v-modal { + opacity: 0; + width: 0; + height: 0; + } + + .close-confirm-box { + width: 420px; + } + + .close-confirm-box__content { + display: flex; + margin-top: -10px; + margin-bottom: -10px; + } + + .close-confirm-box__status { + color: $yellow-color; + font-size: 36px !important; + } + + .close-confirm-box__message { + line-height: 36px; + padding-left: 8px; + } + + .grayBg { + background: $text-over-color; + color: $body-background; + padding: 5px 9px; + font-size: 12px; + border-radius: 2px; + border: none; + margin-left: 10px; + border: $line-border; + } + + .el-tooltip__popper.is-light { + border-color: $line-color; + color: #3d3d3d; + } +} diff --git a/web/src/apps/scriptis/components/uploadDialog/index.vue b/web/src/apps/scriptis/components/uploadDialog/index.vue new file mode 100755 index 000000000..2ced47d51 --- /dev/null +++ b/web/src/apps/scriptis/components/uploadDialog/index.vue @@ -0,0 +1,199 @@ + + + diff --git a/web/src/apps/scriptis/i18n/common/en.json b/web/src/apps/scriptis/i18n/common/en.json new file mode 100755 index 000000000..faec8f555 --- /dev/null +++ b/web/src/apps/scriptis/i18n/common/en.json @@ -0,0 +1,812 @@ +{ + "message": { + "scripts": { + "emptyText": "No data yet", + "publishing": "Publishing", + "tagPlaceholder": "tag，click enter add", + "script": "Script", + "ok": "OK", + "editor": "Edit", + "failedNotice": "Invalid items found, please check and then retry!", + "cancel": "Cancel", + "notSave": "Cannot save on read-only status!", + "saveSuccess": "Successfully saved!", + "saveErr": "Save failed!", + "createdTitle": "Table creation guide", + "saveToWidget": "Save widget to solution", + "contextMenu": { + "openToTab": "Open as a tab", + "copyPath": "Copy path", + "addCatalog": "New directory", + "addScript": "New script", + "importToHive": "Import to Hive", + "importToHdfs": "Import to HDFS", + "copyscript": "Copy script", + "stickscript": "Stick script" + }, + "warning": { + "getRootPath": "Failed to request Workspace root directory, please refresh to retry!", + "scriptChanged": "Script already updated, please save and retry!", + "empty": "Empty script content!" + }, + "error": { + "fileExists": "Duplicated file!" + }, + "boyondQuota": { + "title": "Warning", + "desc": "'You have open 20 scripts(over maximun limit), please close some other scripts!'" + }, + "createTable": { + "action": "Action", + "delete": "Del", + "baseAttr": "Table basic attributes", + "addMore": "Batch Add", + "add": "New", + "partitionAttr": "Partition attributes", + "selectResource": "Please choose the original field", + "next": "Next", + "nameNotice": "Only Chinese characters, alphanumeric characters and underscore are allowed", + "zm": "alphabetic characters", + "daName": "Database name", + "tableName": "Table name", + "tablePlaceholder": "Starts with alphabetic characters; Only alphanumeric characters and underscore are allowed", + "tableOtherName": "Table alias name", + "aliasPlaceholder": "Only Chinese characters, alphanumeric characters and underscore are allowed", + "use": "Usage", + "emptyPlaceholder": "Only Chinese characters, alphanumeric characters and underscore are allowed(Can be empty)", + "product": "Product", + "project": "Project", + "tableDesc": "Table description", + "selectDbName": "Please choose database name", + "inputTableName": "Please enter table name", + "inputOtherName": "Please enter alias name", + "letter": "Alphabetic characters", + "moduleAttr": "Table model attributes", + "partitionType": "Partition type", + "partition": "Partition", + "unartition": "Not Partition", + "lifecycle": "Lifecycle", + "forever": "Forever", + "day": "Valid for a day", + "week": "Valid for a week", + "moon": "Valid for a month", + "midYear": "Valid for half a year", + "moduleRank": "Model layer", + "resource": "Original data layer", + "detailLevel": "Detailed data layer", + "sumData": "Summarized data layer", + "appData": "Application data layer", + "actionType": "Usage method", + "noeWrite": "Write Once Read Many", + "ZSGC": "CRUD", + "FG": "Multi-overwrite", + "OED": "Write Once Read Occasionally", + "isUse": "Is used externally", + "isDwsExeis": "Need to be used in systems outside DsS", + "selectPartitionType": "Please choose partition type", + "selectLife": "Please choose lifecycle", + "selectModuleLevel": "Please choose model layer", + "selectUseType": "Please choose usage method", + "selectIsUse": "Please choose whether to be used externally", + "canned": "Duplicated table, please enter other table names", + "type": "Type", + "selectType": "Please choose type", + "path": "Path", + "FGF": "Separator", + "BMGS": "Encoding format", + "XZBMGS": "Please choose encoding format", + "XDF": "Limited qualifier", + "WXDF": "Unlimited qualifier", + "HXWBT": "Use the first line as table header", + "HQBZDSJ": "Request table column data", + "XZWJLJ": "Please choose file path", + "GXMLDR": "Import from shared directory", + "DR": "Import", + "DH": "Comma", + "FH": "Semicolon", + "ZBF": "Tab", + "KG": "Space", + "SYH": "Double quote", + "DYH": "Single quote(\"\")", + "GLWJ": "Filter directory", + "LYSX": "Source attribute", + "MBZD": "Target column and source column", + "XZK": "Choose database", + "XZB": "Choose table", + "SJSXLY": "Data attribute source", + "DRFS": "Import method", + "XJ": "New", + "FHSYB": "Back to Previous", + "SCDDL": "Generate DDL statement", + "JB": "Create table", + "DDLYJ": "DDL Statement", + "FZZT": "Copy to clipboard", + "DDLYFZ": "DDL statement has been copied to clipboard!", + "fieldsName": "Column name", + "isPrimary": "Primary key", + "fieldsType": "Column type", + "length": "Length/Settings", + "fieldsAlias": "Column alias name", + "alias": "Alias name", + "sourceFields": "Source column", + "rule": "Column rule", + "express": "Column expression", + "comment": "Comments", + "partitionValue": "Partition value", + "partitionAlias": "Partition alias", + "fieldsNameRequire": "Column name cannot be empty", + "fieldsNameRule": "Only alphanumeric characters and underscore are allowed" + }, + "tableDetails": { + "BZBSX": "Table basic attributes", + "BZDXX": "Table column information", + "BTJXX": "Table statistical information", + "DBN": "Database name", + "TN": "Table namee", + "BBM": "Table alias name", + "FQM": "Partitioned table", + "CJYH": "Created By", + "CJSI": "Created At", + "ZHFWSJ": "Last Accessed At", + "BMS": "Table description", + "BMXSX": "Table model attributes", + "SMZQ": "Lifecycle", + "MXCJ": "Model layer", + "SYFS": "Usage mothod", + "WBSFSY": "Is used externally", + "BYYSX": "Table usage attributes", + "SXCP": "Product", + "SSXM": "Project", + "YT": "Usage", + "SSZDMC": "Search for column name...", + "XH": "Index", + "ZDM": "Column name", + "ZDLX": "Column type", + "BM": "Alias", + "SFZJ": "Is primary key", + "SFFQ": "Is partition", + "ZDGZ": "Column rules", + "MS": "Description", + "FQXX": "Partition information", + "DJFQCK": "Please click the root partition to view size information", + "WFQSJ": "No data of table partitions", + "BXX": "Table information", + "BZD": "Number of columns", + "BDX": "Table size", + "WJM": "Number of files", + "FQS": "Number of partitions", + "DX": "size", + "FQDX": "Partition size:", + "WJS": "Number of files:", + "YES": "Yes", + "GSHJX": "Formatted to", + "ZJXYGE": "The format required by the component" + }, + "logView": { + "taskId": "Task ID：", + "search": "Search", + "filter": "Filter", + "error": "Error", + "inputTaskId": "Please enter a valid task ID!" + }, + "functionSetting": { + "HSM": "Function name", + "HSLX": "Funtion type", + "TY": "Common", + "ZDYHS": "Self-defined funtions", + "B": "Package", + "JBLU": "Script path", + "ZCGS": "Register format", + "ZCHS": "Register function", + "FHLX": "Return type", + "CSLX": "Parameter type", + "YLLX": "Preview", + "SYGS": "Using format", + "SRLX": "Input type", + "HSMS": "Funtion description", + "MRJZ": "Loaded by default", + "XZHS": "New function", + "LJCJ": "Create", + "QSRMC": "Please enter the name", + "CDZ": "Length should be between 1 to 100 characters", + "BXYZMKT": "Must starts with alphabetic characters, followed by only alphanumeric, underscore and Chinese characters", + "QZSXZYZLX": "Please choose at least one type", + "QSRWZLJ": "Please enter the complete path", + "KHDBJZPP": "Brackets indicate precise matching", + "HZMZC": "Only jar is allowed as the suffix. Only alphanumeric characters and underscore is allowed for the name of jar package!", + "ZCPYSCA": "Only py and scala are allowed as the suffix!", + "BHJZCYW": "Only alphabetic characters are allowed as the package name", + "XHX": "Underscore", + "SRZCHS": "Please enter function registration", + "SRFHLX": "Please enter the return type", + "SRCSLX": "Please enter the parameter type", + "XGHS": "Update funtion", + "XG": "Update" + }, + "functionShare": { + "ZZGX": "Sharing function {name} in progress", + "HSLX": "Funtion type", + "GXYH": "Shared users", + "SRGXYH": "Please enter the users to be shared, separated by comma", + "XZGXHSLX": "Please choose a type for this shared function!", + "ZSXZYWGXYH": "Please enter at least one user needed to be shared with!" + }, + "uploadDialog": { + "SCJBWJ": "Upload script file", + "TWJDSQ": "Drag file here, or click to upload ", + "PATH": "Path:", + "QKLB": "Clear list", + "LIMIT2M": "Script file to upload cannot exceed 2MB!", + "LIMIT100M": "Script file to upload cannot exceed 100MB!", + "ZDSC": "This operation will interrupt file uploading, continue or not? ", + "TS": "Notice", + "ZJGB": "Close directly", + "QX": "Cancel", + "QXBBSCCK": "Close uploading window", + "SCCG": "Successfully uploaded file {name}!", + "WJDXCCXE": "File size exceeds limit", + "WJYCZ": "Duplicated file, please choose another file or directory!", + "WJMBHF": "Invalid file name, only alphanumeric characters, Chinese characters and underscore is allowed with suffix!", + "SCZQK": "Uploading file in progress, cannot clear upload list!" + }, + "importToHive": { + "MC": "Name", + "ZDMC": "Column name cannot be empty", + "JZCZM": "Only alphanumeric characters and underscore are allowed", + "QSRZDM": "Please enter column name", + "LX": "Type", + "GS": "Format", + "QXZGS": "Please choose date format", + "CD": "Length", + "JD": "Accuracy", + "XSW": "Decimal scale", + "SRZDZS": "Please enter column comments", + "DRWJ": "Import file", + "BZONE": "Step 1", + "BZTOW": "Step 2", + "WJDR": "Import data from file", + "DRZMRB": "Import to the default table", + "LY": "Source", + "QXZLX": "Please choose type", + "LJ": "Path", + "GSH": "Format", + "FGF": "Separator", + "BMGS": "Encoding format", + "QXZBMGS": "Please choose encoding format", + "XDF": "Qualifier", + "WXDF": "No qualifier", + "SHWBT": "Treat the first line as header", + "SJK": "Database name", + "QSRKM": "Please enter database name", + "SJB": "Table name", + "SRSJBM": "Please enter table name", + "FQ": "Partition", + "FX": "Overwrite", + "XZFQ": "New partition", + "SHEETB": "sheet table", + "ZG": "Column", + "XYB": "Next step", + "SYB": "Previous step", + "TJ": "Submit", + "LXWCHAR": "Column with type char or varchar should have length as an integer greater than 0", + "ZDJDSRYW": "Column with type decimal has invalid accuracy", + "ZDXSWYDY": "Column with type decimal should have scale as an integer greater than 0", + "ZDZSZCZW": "Only Chinese characters, alphabetic characters and underscore are allowed in column comments, with length not greater than 100 characters", + "SJKMBNWK": "Database name cannot be empty", + "BMBNWK": "Table name cannot be empty", + "BMJZCZMKT": "Table names should start with alphabetic characters, followed by alphabetic characters and underscore, length between 0 to 100 characters", + "FQMBNWK": "Partition name cannot be empty", + "FQMZCSZ": "Only alphanumeric characters and - are allowed in partition names", + "FQZBNWK": "Partition value cannot be empty", + "FQZZZCSZ": "Only numbers are allowed as partition values", + "GXMLDR": "Import from shared directory", + "HDFSDR": "Import from HDFS", + "DH": "Comma(,)", + "FH": "Semicolon(;)", + "ZBF": "Tab(\\t)", + "KG": "Space", + "SYH": "Double quote", + "DYH": "Single quote", + "WJZZZXDR": "Importing file in progress, please retry later", + "FQZCF": "Duplicated partition values, please confirm whether to overwrite", + "TSWST": "Notice: Cannot handle view tables, please choose other tables to import", + "WFCZSTB": "Cannot handle view tables", + "QXZQTBDR": "Please choose other tables to import", + "BMYCZDQSJK": "Table name already exists in this database", + "ZBZCDJFQ": "Multi-level partitions are not supported yet, please choose other tables", + "QQURFQ": "Please confirm the partition", + "HQFQSB": "Failed to request partition information", + "ZQDRWJLJ": "Please choose the path correctly", + "QQRSJKMCXX": "Please confirm the database name", + "QQRSFCF": "Please confirm whether to overwrite" + }, + "hiveTableExport": { + "DCB": "Export form table", + "SZDCCS": "Step 1: Set export parameters", + "SZDCLJ": "Step 2: Set export path", + "SJKM": "Database name", + "QSRKM": "Please enter the database name", + "SJBM": "Table name", + "ZDXX": "Column information", + "FQXX": "Partition information", + "DCGS": "Export format", + "FGF": "Separator", + "BMGS": "Encoding format", + "XZBMGS": "Please choose the encoding format", + "KZTH": "Null value place holder", + "SXWBT": "Treat the first line as header", + "LX": "Type", + "QXZLX": "Please choose the type", + "DCZ": "Export to", + "WJM": "File name", + "TRDCWJMC": "Please enter the name of the file to be exported", + "FXMS": "Overwrite mode", + "ZJ": "Append", + "FX": "Overwrite", + "BM": "sheet table name", + "QSR": "Please enter", + "XYB": "Next step", + "SYB": "Previous step", + "TJ": "Submit", + "WFCZST": "Cannot handle view tables, please choose other tables to export", + "ZBZCDBDC": "Tables with multiple partitions are not supported yet, please choose other tables to export", + "GXML": "Shared directory", + "DH": "Comma(,)", + "FH": "Semicolon(;)", + "ZBF": "Tab(\\t)", + "KG": "Space", + "KZFC": "Empty string", + "XZSJK": "Please choose database", + "XZB": "Please choose table", + "XZFGF": "Please choose separator", + "XZYGFQBDC": "Please choose a partitioned table to export", + "ZSXZYGZD": "Please choose at least one column", + "XZDCGS": "Please choose export format", + "XZXYJKZTHDZ": "Please choose the Null value place holder", + "XZDCLJDLX": "Please choose the type of export path", + "XZDCLJ": "Please choose export path", + "DRDCDWJ": "Please enter the name of the file to be exported", + "CDZ": "Length should be between 3 to 100 characters", + "WJMZZCZW": "Only Chinese characters, alphanumeric characters and underscore are allowed", + "CBDC": "Exporting table in progress, please hold on or try other tables", + "QSHZSS": "Please hold on or try other tables", + "XZFQ": "Please choose partition", + "BDXCG": "The size of this table is greater than 5GB, please choose other methods to export or choose other tables" + }, + "hiveTableDesc": { + "searchName": "Search by column name...", + "tableDetail": "Table details", + "noData": "No data for table partition information yet", + "tableColumns": "Table columns", + "columnsName": "Column name", + "columnsType": "Column type", + "tablePartitions": "Table partitions表分区", + "dbName": "Database name", + "tableName": "Table name", + "createUser": "Created By", + "createTtime": "Created At", + "lastTime": "Last accessed", + "tableSize": "Table size", + "size": "Size" + }, + "deleteType": { + "function": "Function", + "folder": "Directory", + "file": "File", + "common": "Common", + "catalog": "Directory", + "script": "Script file", + "engine": "Engine", + "task": "Task", + "engineAndTask": "Engine and task" + }, + "constants": { + "add": "New", + "cancel": "Cancel", + "confirm": "Confirm", + "changeLanguage": "Switch language", + "detail": "Details", + "description": "Description", + "delete": "Delete", + "download": "Download", + "edit": "Edit", + "export": "Export", + "fail": "Fail", + "noDataText": "No data yet", + "node": "Node", + "refresh": "Refresh", + "rename": "Rename", + "reset": "Reset", + "remove": "Remove", + "submit": "Submit", + "search": "Search", + "save": "Save", + "upload": "Upload", + "update": "Update", + "warning": { + "api": "Requesting API, please hold on!", + "data": "Requesting data, please hold on!", + "waiting": "Please wait for API response!", + "biLoading": "Connecting with Visualis, please hold on!", + "comingSoon": "New version is being integrated, so stay tuned !", + "GJZ": "Open source co-construction is in progress, so stay tuned!" + }, + "success": { + "delete": "Successfully deleted!", + "update": "Successfully updated!", + "add": "Successfully created!", + "save": "Successfully saved", + "stick": "Sticking success", + "prohibit": "Prohibit copying this type" + }, + "error": { + "refresh": "Error on refreshing, please retry!", + "fileExists": "Duplicated file", + "folderExists": "Duplicated directory", + "functionExists": "Duplicated function" + }, + "logPanelList": { + "fullScreen": "Full Screen", + "releaseFullScreen": "Restore", + "min": "Minimize", + "releaseMin": "Restore" + }, + "time": { + "second": "Second", + "minute": "Minute", + "hour": "Hour", + "day": "Day" + } + }, + "navList": { + "dataStudio": "Data development", + "workSpace": "Workspace", + "database": "Database", + "udfFunction": "UDF", + "methodFunction": "Method", + "hdfs": "HDFS" + }, + "newDialog": { + "catalog": "Directory", + "scriptName": "Name", + "catalogName": "Name", + "targetScriptPath": "Creation target path", + "placeholder": "Please enter directory name", + "required": "Please enter directory name", + "rules": { + "scriptName": { + "required": "Please enter script name", + "lengthLimit": "Length should be between 1 to 200 characters", + "letterTypeLimit": "Only Chinese characters, alphanumeric characters and underscore are allowed" + }, + "catalogName": { + "required": "Please enter directory name", + "lengthLimit": "Length should be between 1 to 200 characters", + "letterTypeLimit": " Only Chinese characters, alphanumeric characters and underscore are allowed" + }, + "targetScriptPath": { + "required": "Please choose target script path" + } + }, + "nodeType": "Node type", + "scriptType": "Script type", + "targetFolder": "Target directory" + }, + "database": { + "contextMenu": { + "db": { + "copyName": "Copy database name", + "pasteName": "Copy database name & paste to script", + "listSort": "Sort by time", + "liststringSort": "Sort by first character" + }, + "tb": { + "queryTable": "Query table", + "deleteTable": "Drop table", + "viewTable": "View table structure", + "exportTable": "Export table", + "copyName": "Copy table name", + "pasteName": "Copy table name & paste to script", + "copyAllColumns": "Copy all the columns of table" + }, + "field": { + "copyColumnsName": "Copy column name", + "pasteName": "Copy column name & paste to script" + } + }, + "partitionsTitle": "Partition size", + "notPosition": "Failed to locate last operation, please use refresh after right-click" + }, + "fnSideBar": { + "contextMenu": { + "isOwn": { + "addFunction": "New function", + "addFolder": "New directory", + "share": "Share" + }, + "isLeaf": { + "copyName": "Copy function name", + "pasteName": "Copy function name & paste to script" + }, + "isOwnAndLeaf": { + "updateToExpired": "Set as expired", + "viewSharedUsers": "View shared users" + } + }, + "warning": { + "folderNameExists": "Duplicated directory name", + "functionNameExists": "Duplicated function name", + "functionShared": "This function has been shared, please set it as expired before deletion!", + "noEmpty": "Please manually clear sub directories/functions and then retry!" + }, + "success": { + "load": "Successfully loaded UDF!", + "cancelLoading": "Canceled loading UDF!", + "addUdf": "Successfully created UDF!", + "addFolder": "Successfully created new directory!", + "shareFunction": "Successfully shared function", + "updateShareUser": "Successfully updated shared users!", + "functionExpired": "Successfully set function as expired, please view it in Shared Functions!" + }, + "confirm": { + "title": "Notice", + "content": "

This operation will affect all the shared users, continue or not?

" + } + }, + "hdfs": { + "contextMenu": { + "openToTab": "Open as a tab", + "copyPath": "Copy path", + "addCatalog": "New directory", + "importToHive": "Export to Hive", + "exportToWorkSpace": "Export to Workspace" + }, + "dialogTitle": { + "exportToWorkSpace": "Export to Workspace" + }, + "warning": { + "noRootPath": "Failed to request HDFS root directory, please refresh to retry!" + } + }, + "container": { + "title": { + "editorName": "Editor" + }, + "closeDropDown": { + "others": "Close Others", + "all": "Close All", + "left": "Close Left", + "right": "Close Right" + }, + "closeHint": "Close Hint", + "footer": { + "cancel": "Cancel", + "close": "Close", + "save": "Save", + "saveAs": "Save As" + }, + "tips": "What is Scriptis？\nScriptis is a one-stop interactive data analysis & exploration tool built by WeDataSphere at Webank. Using Linkis as its kernel, Scriptis provides functionalities including multiple computation engines(Spark, Hive, TiSpark...), Hive database management, resource management(such as Yarn resource and server resource), application management and user resource management(such as UDF and variables). \nScriptis？\n1. Select a directory in workspace, and create a sub directory; \n2. Right-click a directory => Create script; \n3. Select script type, for example: SQL, Pyspark, HQL, etc... \n4. Edit script, click & run, wait for result sets.", + "warning": { + "noDBInfo": "Failed to request database/table information, word association in script may not be working well! Please refresh to retry!", + "noUDF": "Failed to request UDF and Method information, word association in script may not be working well! Please refresh to retry!", + "noSelectedScript": "No script selected, please select one first and retry!" + }, + "notice": { + "developerWarning": { + "title": "Warning", + "desc": "Warning from developers: Requesing add API must pass id and filename as parameters!" + }, + "unSupport": { + "title": "Warning", + "desc": "Sorry, the system does not support this file format!" + } + }, + "removeWork": { + "normal": "This script has been updated, save or not?", + "readOnly": "This read-only script has been updated, save it as another file to workspace or not? ", + "temporary": "This temporary script has been updated, save it as another file or not?" + } + }, + "editorDetail": { + "navBar": { + "undo": "Undo", + "redo": "Redo", + "play": "Run", + "stop": "Stop", + "save": "Save", + "config": "Configuration" + }, + "warning": { + "running": "Script execution in progress, please hold on!", + "invalidArgs": "Please check the correctness of the input of configuration parameters!", + "emptyCode": "Cannot execute empty script!" + } + }, + "history": { + "columns": { + "taskID": { + "title": "Task ID" + }, + "runningTime": { + "title": "Execution Time" + }, + "createDate": { + "title": "Executed At" + }, + "status": { + "title": "Status", + "status": { + "succeed": "Succeeded", + "running": "Running", + "timeout": "Timeout", + "inited": "Waiting", + "scheduled": "Applying Resources", + "failed": "Failed", + "cancel": "Canceled" + } + }, + "data": { + "title": "Code" + }, + "failedReason": { + "title": "Key information" + }, + "control": { + "title": "Handle", + "view": "View", + "download": "Download log" + } + }, + "success": { + "open": "Successfully open", + "download": "Successfully downloaded, please check from local download directory!" + } + }, + "saveAs": { + "header": "Save as script", + "formItems": { + "fileName": { + "label": "File name" + }, + "selectedPath": { + "label": "Choose path" + } + }, + "rules": { + "fileName": { + "required": "Please enter the file name", + "lengthLimit": "Length between 1 to 200 characters", + "letterTypeLimit": "Only Chinese characters, alphanumeric characters and underscore are allowed" + }, + "selectedPath": { + "required": "Please choose a directory" + } + }, + "warning": { + "invalid": "Please check the items failed to pass validations!" + }, + "success": { + "saveScript": "Successfully saved as script, please refresh directory tree manually!" + } + }, + "tabs": { + "progress": "Progress", + "result": "Execution Result", + "log": "Execution Log", + "history": "History", + "diagnosis": "Intelligent Diagnosis" + }, + "notice": { + "unsave": { + "title": "Notice", + "desc": "Your temporary script has been updated, please mind to save!" + }, + "sendStart": { + "title": "Execution Notice", + "render": "Start to execute the script" + }, + "querySuccess": { + "title": "Execution result notice", + "render": "Successfully executed script, time elapsed:" + }, + "notice": { + "title": "Execution Result Notice" + }, + "kill": { + "title": "Execution Notice", + "desc": "Stopped executing this script" + } + }, + "confirmModal": { + "title": "Save as", + "content": "

This temporary has been updated!

Save it or not?

", + "okText": "Save As", + "cancelText": "Cancel" + }, + "setting": { + "header": "Settings", + "customVariable": { + "title": "Customized Variables" + }, + "envVariable": { + "title": "Environment Variables" + }, + "runTimeArgs": { + "settingTitle": "Command line Arguments", + "placeholder": "Please use space to separate multiple arguments" + } + }, + "apiPublish": { + "title": "PublishAPI", + "button": { + "addApi": "API Add", + "updateApi": "API Update" + }, + "addApiModal": { + "modalTitle": "Add API", + "contentTitle": "Base Info", + "paramConfirmTitle": "Params Confirm", + "apiName": "API Name", + "apiPath": "API Path", + "protocol": "Protocol", + "requestType": "Request Type", + "tag": "Tag", + "visible": "Visible", + "describe": "Describe", + "nextStep": "Next", + "back": "Back", + "cancel": "Cancel", + "approvalName": "Approval Name", + "applyUser": "Apply User", + "proxyUser": "Proxy User", + "confirm": "Confirm", + "comment": "Comment" + }, + "updateApiModal": { + "modalTitle": "Update API", + "apiVersionUpgrade": "New Version", + "paramConfirm": "Please confirm param" + }, + "paramTable": { + "paramName": "Param Name", + "paramType": "Param Type", + "require": { + "title": "Require", + "yes": "Yes", + "no": "No", + "hide": "Hide" + }, + "defaultValue": "Default Value", + "displayName": "Display Name", + "describe": "Describe" + }, + "rule": { + "nameRule": "Please input api name!", + "approvalName": "Please input name!", + "pathRule": "Please input api path!", + "pathRegRule": "Api path illegal!", + "requestTypeRule": "Please select request type!", + "protocolRule": "Please select protocol!", + "valueRule": "Started with alphabetic characters, only alphanumeric and underscore are allowed!", + "contentLengthLimit": "Length limit 155", + "contentLengthLimit50": "Length limit 50", + "contentLengthLimitTwo": "Length limit 200", + "pathRepeat": "Api path exists", + "nameRepeat": "Api name exists", + "chineseCheck": "Chainese only", + "chineseName": "Chinese name of API" + }, + "visible": { + "workspace": "Workspace", + "personal": "Personal", + "public": "Public", + "grantView": "Grant View" + }, + "notice": { + "publishSuccess": "Publish Success" + }, + "tagPlaceholder": "tag，click enter add" + } + } + } + } diff --git a/web/src/apps/scriptis/i18n/common/zh.json b/web/src/apps/scriptis/i18n/common/zh.json new file mode 100755 index 000000000..8e0181f15 --- /dev/null +++ b/web/src/apps/scriptis/i18n/common/zh.json @@ -0,0 +1,812 @@ +{ + "message": { + "scripts": { + "emptyText": "暂无数据", + "publishing": "发布中", + "tagPlaceholder": "标签，按 enter 创建", + "script": "脚本", + "editor": "修改", + "ok": "确认", + "failedNotice": "验证项未通过，请检查后再试！", + "cancel": "取消", + "notSave": "只读状态不可保存！", + "saveSuccess": "保存成功！", + "saveErr": "保存失败！", + "createdTitle": "建表向导", + "saveToWidget": "保存widget到解决方案", + "contextMenu": { + "openToTab": "打开到侧边", + "copyPath": "复制路径", + "addCatalog": "新建目录", + "addScript": "新建脚本", + "importToHive": "导入至Hive", + "importToHdfs": "导入至HDFS", + "copyscript": "复制脚本", + "stickscript": "粘贴" + }, + "warning": { + "getRootPath": "未请求到工作空间根目录，请刷新重试！", + "scriptChanged": "脚本已发生改变，请保存后再试！", + "empty": "脚本内容为空！" + }, + "error": { + "fileExists": "该文件已经存在！" + }, + "boyondQuota": { + "title": "警告", + "desc": "'您已打开20个脚本（超出最大限额），请关闭其他脚本再打开！'" + }, + "createTable": { + "action": "操作", + "delete": "删除", + "baseAttr": "表基本属性", + "addMore": "批量新增", + "add": "新增", + "partitionAttr": "分区属性", + "selectResource": "请选择来源字段", + "next": "下一步", + "nameNotice": "仅支持中文、字母、数字和下划线", + "zm": "字母", + "daName": "数据库名", + "tableName": "表名", + "tablePlaceholder": "字母开头，支持字母、数字和下划线", + "tableOtherName": "表别名", + "aliasPlaceholder": "支持中文、字母、数字和下划线", + "use": "用途", + "emptyPlaceholder": "支持中文、字母、数字和下划线(可为空)", + "product": "所属产品", + "project": "所属项目", + "tableDesc": "表描述", + "selectDbName": "请选择库名", + "inputTableName": "请输入表名", + "inputOtherName": "请输入别名", + "letter": "大小写字母", + "moduleAttr": "表模型属性", + "partitionType": "分区类型", + "partition": "分区", + "unartition": "非分区", + "lifecycle": "生命周期", + "forever": "永久", + "day": "当天有效", + "week": "一周有效", + "moon": "一月有效", + "midYear": "半年有效", + "moduleRank": "模型层级", + "resource": "原始数据层", + "detailLevel": "明细数据层", + "sumData": "汇总数据层", + "appData": "应用数据层", + "actionType": "使用方式", + "noeWrite": "一次写多次读", + "ZSGC": "增删改查", + "FG": "多次覆盖写", + "OED": "一次写偶尔读", + "isUse": "外部是否使用", + "isDwsExeis": "是否需要在dws之外的系统上使用", + "selectPartitionType": "请选择分区类型", + "selectLife": "请选择生命周期", + "selectModuleLevel": "请选择模型层级", + "selectUseType": "请选择使用方式", + "selectIsUse": "请选择外部是否使用", + "canned": "该表已存在，请输入其他表名", + "type": "类型", + "selectType": "请选择类型", + "path": "路径", + "FGF": "分隔符", + "BMGS": "编码格式", + "XZBMGS": "请选择编码格式", + "XDF": "限定符", + "WXDF": "无限定符", + "HXWBT": "首行为表头", + "HQBZDSJ": "获取表字段数据", + "XZWJLJ": "请选择文件路径", + "GXMLDR": "共享目录导入", + "DR": "导入", + "DH": "逗号", + "FH": "分号", + "ZBF": "制表符", + "KG": "空格", + "SYH": "双引号", + "DYH": "单引号", + "GLWJ": "过滤文件夹", + "LYSX": "来源属性", + "MBZD": "目标表字段与源字段", + "XZK": "选择库", + "XZB": "选择表", + "SJSXLY": "数据属性来源", + "DRFS": "导入方式", + "XJ": "新建", + "FHSYB": "返回上一步", + "SCDDL": "生成DDL语句", + "JB": "建表", + "DDLYJ": "DDL语句", + "FZZT": "复制至粘贴板", + "DDLYFZ": "DDL语句已复制至粘贴板！", + "fieldsName": "字段名", + "isPrimary": "主键", + "fieldsType": "字段类型", + "length": "长度/设置", + "fieldsAlias": "字段别名", + "alias": "别名", + "sourceFields": "来源字段", + "rule": "字段规则", + "express": "字段表达式", + "comment": "描述", + "partitionValue": "分区值", + "partitionAlias": "分区别名", + "fieldsNameRequire": "字段名称不能为空", + "fieldsNameRule": "仅支持字母数字和下划线" + }, + "tableDetails": { + "BZBSX": "表基本属性", + "BZDXX": "表字段信息", + "BTJXX": "表统计信息", + "DBN": "数据库名", + "TN": "表名", + "BBM": "表别名", + "FQM": "分区表", + "CJYH": "创建用户", + "CJSI": "创建时间", + "ZHFWSJ": "最后访问时间", + "BMS": "表描述", + "BMXSX": "表模型属性", + "SMZQ": "生命周期", + "MXCJ": "模型层级", + "SYFS": "使用方式", + "WBSFSY": "外部是否使用", + "BYYSX": "表应用属性", + "SXCP": "所属产品", + "SSXM": "所属项目", + "YT": "用途", + "SSZDMC": "搜索字段名称...", + "XH": "序号", + "ZDM": "字段名", + "ZDLX": "字段类型", + "BM": "别名", + "SFZJ": "是否主键", + "SFFQ": "是否分区", + "ZDGZ": "字段规则", + "MS": "描述", + "FQXX": "分区信息", + "DJFQCK": "请点击根分区查看分区大小", + "WFQSJ": "暂无表分区数据", + "BXX": "表信息", + "BZD": "字段数", + "BDX": "表大小", + "WJM": "文件数", + "FQS": "分区数", + "DX": "大小", + "FQDX": "分区大小：", + "WJS": "文件数：", + "YES": "对", + "GSHJX": "进行格式化成", + "ZJXYGE": "组件需要的格式" + + }, + "logView": { + "taskId": "任务ID：", + "search": "搜索", + "filter": "过滤", + "error": "错误", + "inputTaskId": "请输入有效的任务ID！" + }, + "functionSetting": { + "HSM": "函数名称", + "HSLX": "函数类型", + "TY": "通用", + "ZDYHS": "自定义函数", + "B": "jar包", + "JBLU": "脚本路径", + "ZCGS": "注册格式", + "ZCHS": "注册函数", + "FHLX": "返回类型", + "CSLX": "参数类型", + "YLLX": "预览", + "SYGS": "使用格式", + "SRLX": "输入类型", + "HSMS": "函数描述", + "MRJZ": "默认加载", + "XZHS": "新增函数", + "LJCJ": "立即创建", + "QSRMC": "请输入名称", + "CDZ": "长度在1到100个字符", + "BXYZMKT": "必须以字母开头，且只支持字母、数字、下划线和中文", + "QZSXZYZLX": "请至少选择一种类型", + "QSRWZLJ": "请输入完整的路径", + "KHDBJZPP": "括号代表精准匹配", + "HZMZC": "后缀名只支持jar，且jar包名仅支持英文，数字，下划线！", + "ZCPYSCA": "后缀名只支持py、scala！", + "BHJZCYW": "包名仅支持英文", + "XHX": "下划线", + "SRZCHS": "请输入注册函数", + "SRFHLX": "请输入返回类型", + "SRCSLX": "请输入参数类型", + "XGHS": "修改函数", + "XG": "修改" + }, + "functionShare": { + "ZZGX": "正在对{name}函数进行共享操作", + "HSLX": "函数类型", + "GXYH": "共享用户", + "SRGXYH": "请输入要共享的用户，以逗号分隔", + "XZGXHSLX": "请选择该共享函数的类型！", + "ZSXZYWGXYH": "请填入至少一位需要共享的用户！" + }, + "uploadDialog": { + "SCJBWJ": "上传脚本文件", + "TWJDSQ": "将文件拖到此处，或点击上传", + "PATH": "路径：", + "QKLB": "清空列表", + "LIMIT2M": "上传脚本文件不得超过2MB！", + "LIMIT100M": "上传脚本文件不得超过100MB！", + "ZDSC": "此操作将中断文件上传，是否继续？", + "TS": "提示", + "ZJGB": "直接关闭", + "QX": "取消", + "QXBBSCCK": "取消关闭上传窗口", + "SCCG": "文件{name}上传成功！", + "WJDXCCXE": "文件大小超出限额", + "WJYCZ": "文件已存在，请选择其它文件或选择其它文件夹！", + "WJMBHF": "文件名称不合法，仅支持以字母、数字、中文、下划线且带后缀的命名！", + "SCZQK": "文件上传中，无法清空上传列表！" + }, + "importToHive": { + "MC": "名称", + "ZDMC": "字段名称不能为空", + "JZCZM": "仅支持字母数字和下划线", + "QSRZDM": "请输入字段名", + "LX": "类型", + "GS": "格式", + "QXZGS": "请选择日期格式", + "CD": "长度", + "JD": "精度", + "XSW": "小数位数", + "SRZDZS": "请输入字段注释", + "DRWJ": "导入文件", + "BZONE": "步骤1", + "BZTOW": "步骤2", + "WJDR": "从文件导入数据", + "DRZMRB": "导入至默认表", + "LY": "来源", + "QXZLX": "请选择类型", + "LJ": "路径", + "GSH": "格式化", + "FGF": "分隔符", + "BMGS": "编码格式", + "QXZBMGS": "请选择编码格式", + "XDF": "限定符", + "WXDF": "无限定符", + "SHWBT": "首行为表头", + "SJK": "数据库名", + "QSRKM": "请输入库名", + "SJB": "数据表名", + "SRSJBM": "请输入数据表名", + "FQ": "分区", + "FX": "复写", + "XZFQ": "新增分区", + "SHEETB": "sheet表", + "ZG": "字段", + "XYB": "下一步", + "SYB": "上一步", + "TJ": "提交", + "LXWCHAR": "类型为char或者varchar的字段长度应为大于0的正整数", + "ZDJDSRYW": "类型为decimal的字段精度输入有误", + "ZDXSWYDY": "类型为decimal的字段小数位数应为大于0的正整数", + "ZDZSZCZW": "字段注释仅支持中文、大小写字母和下划线，长度不得超过100字符", + "SJKMBNWK": "数据库名不能为空", + "BMBNWK": "表名不能为空", + "BMJZCZMKT": "表名仅支持以字母开头，名称包含大小写字母和下划线，长度为0~100", + "FQMBNWK": "分区名不能为空", + "FQMZCSZ": "分区名只支持数字、字母和-", + "FQZBNWK": "分区值不能为空", + "FQZZZCSZ": "分区值只支持数字", + "GXMLDR": "共享目录导入", + "HDFSDR": "HDFS导入", + "DH": "逗号(,)", + "FH": "分号(;)", + "ZBF": "制表符(\\t)", + "KG": "空格", + "SYH": "双引号", + "DYH": "单引号", + "WJZZZXDR": "文件正在执行导入，请稍后再试", + "FQZCF": "分区值重复，请确认是否复写", + "TSWST": "提示：无法操作视图表， 请选择其它表导入", + "WFCZSTB": "无法操作视图表", + "QXZQTBDR": "请选择其它表导入", + "BMYCZDQSJK": "表名已存在当前数据库", + "ZBZCDJFQ": "暂不支持多级分区，请选择其它表", + "QQURFQ": "请确认分区", + "HQFQSB": "获取分区信息失败", + "ZQDRWJLJ": "请正确选择导入文件路径", + "QQRSJKMCXX": "请确认数据库名称填写信息", + "QQRSFCF": "请确认是否复写" + }, + "hiveTableExport": { + "DCB": "导出表", + "SZDCCS": "步骤1：设置导出参数", + "SZDCLJ": "步骤2：设置导出路径", + "SJKM": "数据库名", + "QSRKM": "请输入库名", + "SJBM": "数据表名", + "ZDXX": "字段信息", + "FQXX": "分区信息", + "DCGS": "导出格式", + "FGF": "分隔符", + "BMGS": "编码格式", + "XZBMGS": "请选择编码格式", + "KZTH": "空值替换", + "SXWBT": "首行为表头", + "LX": "类型", + "QXZLX": "请选择类型", + "DCZ": "导出至", + "WJM": "文件名", + "TRDCWJMC": "请填入导出文件名称", + "FXMS": "复写模式", + "ZJ": "追加", + "FX": "复写", + "BM": "sheet表名", + "QSR": "请填入", + "XYB": "下一步", + "SYB": "上一步", + "TJ": "提交", + "WFCZST": "无法操作视图表，请选择其它表导出", + "ZBZCDBDC": "暂不支持多分区表导出，请选择其它表导出", + "GXML": "共享目录", + "DH": "逗号(,)", + "FH": "分号(;)", + "ZBF": "制表符(\\t)", + "KG": "空格", + "KZFC": "空字符串", + "XZSJK": "请选择数据库", + "XZB": "请选择表", + "XZFGF": "请选择分隔符", + "XZYGFQBDC": "请选择一个分区表导出", + "ZSXZYGZD": "请至少选择一个字段", + "XZDCGS": "请选择导出格式", + "XZXYJKZTHDZ": "请选择需要将空值替换的值", + "XZDCLJDLX": "请选择导出路径的类型", + "XZDCLJ": "请选择导出路径", + "DRDCDWJ": "请填入导出的文件名称", + "CDZ": "长度在3~100个字符", + "WJMZZCZW": "文件名只支持中文、字母、数字和下划线", + "CBDC": "此表导出中，请稍后再试或者选择其他表", + "QSHZSS": "请稍后再试或者选择其他表", + "XZFQ": "请选择分区", + "BDXCG": "表大小超过5GB，请使用其它方式导出或选择其它表" + }, + "hiveTableDesc": { + "searchName": "搜索字段名称...", + "tableDetail": "表详情", + "noData": "暂无表分区数据", + "tableColumns": "表字段", + "columnsName": "字段名", + "columnsType": "字段类型", + "tablePartitions": "表分区", + "dbName": "数据库名", + "tableName": "数据表名", + "createUser": "创建用户", + "createTtime": "创建时间", + "lastTime": "最后访问时间", + "tableSize": "表大小", + "size": "大小" + }, + "deleteType": { + "function": "函数", + "folder": "文件夹", + "file": "文件", + "common": "通用", + "catalog": "目录", + "script": "脚本文件", + "engine": "引擎", + "task": "任务", + "engineAndTask": "引擎和任务" + }, + "constants": { + "add": "新建", + "cancel": "取消", + "confirm": "确认", + "changeLanguage": "切换语言", + "detail": "详情", + "description": "描述", + "delete": "删除", + "download": "下载", + "edit": "编辑", + "export": "导出", + "fail": "失败", + "noDataText": "暂无数据", + "node": "节点", + "refresh": "刷新", + "rename": "重命名", + "reset": "重置", + "remove": "移除", + "submit": "确定", + "search": "搜索", + "save": "保存", + "upload": "上传", + "update": "修改", + "warning": { + "api": "接口请求中，请稍候！", + "data": "数据请求中，请稍候！", + "waiting": "请等待接口返回！", + "biLoading": "正在和Visualis系统通讯，请稍候！", + "comingSoon": "尚未开源，敬请期待！" + }, + "success": { + "delete": "删除成功！", + "update": "修改成功！", + "add": "新建成功！", + "save": "保存成功", + "stick": "粘贴成功", + "prohibit": "禁止复制此类型" + }, + "error": { + "refresh": "刷新出错，请重试！", + "fileExists": "该文件已经存在", + "folderExists": "该文件夹已经存在", + "functionExists": "该函数已经存在" + }, + "logPanelList": { + "fullScreen": "全屏", + "releaseFullScreen": "取消全屏", + "min": "最小化", + "releaseMin": "取消最小化" + }, + "time": { + "second": "秒", + "minute": "分钟", + "hour": "小时", + "day": "天" + } + }, + "navList": { + "dataStudio": "数据开发", + "workSpace": "工作空间", + "database": "数据库", + "udfFunction": "UDF函数", + "methodFunction": "方法函数", + "hdfs": "HDFS" + }, + "newDialog": { + "catalog": "目录", + "scriptName": "名称", + "catalogName": "名称", + "targetScriptPath": "创建路径", + "placeholder": "请输入脚本名称", + "required": "请输入目录名称", + "rules": { + "scriptName": { + "required": "请输入脚本名称", + "lengthLimit": "长度在1到200个字符", + "letterTypeLimit": "仅支持中文、大小写字母、数字和下划线" + }, + "catalogName": { + "required": "请输入目录名称", + "lengthLimit": "长度在1到200个字符", + "letterTypeLimit": "仅支持中文、大小写字母、数字和下划线" + }, + "targetScriptPath": { + "required": "请选择目标脚本路径" + } + }, + "nodeType": "节点类型", + "scriptType": "脚本类型", + "targetFolder": "目标文件夹" + }, + "database": { + "contextMenu": { + "db": { + "copyName": "复制库名", + "pasteName": "复制库名并粘贴至脚本", + "listSort": "表按时间排序", + "liststringSort": "表按首字符排序" + }, + "tb": { + "queryTable": "查询表", + "deleteTable": "删除表", + "viewTable": "查看表结构", + "exportTable": "导出表", + "copyName": "复制表名", + "pasteName": "复制表名并粘贴至脚本", + "copyAllColumns": "复制表全字段" + }, + "field": { + "copyColumnsName": "复制字段名", + "pasteName": "复制库名并粘贴至脚本" + } + }, + "partitionsTitle": "分区大小", + "notPosition": "无法定位上一次操作，请使用右键刷新功能" + }, + "fnSideBar": { + "contextMenu": { + "isOwn": { + "addFunction": "新增函数", + "addFolder": "新增文件夹", + "share": "共享" + }, + "isLeaf": { + "copyName": "复制函数名", + "pasteName": "复制函数并粘贴至脚本" + }, + "isOwnAndLeaf": { + "updateToExpired": "修改为过期", + "viewSharedUsers": "查看已共享的用户" + } + }, + "warning": { + "folderNameExists": "存在同名的文件夹", + "functionNameExists": "存在同名的函数", + "functionShared": "该函数已被共享，请将函数设置为过期再删除！", + "noEmpty": "请手动删除子文件夹或函数后再试！" + }, + "success": { + "load": "加载udf函数成功！", + "cancelLoading": "已取消加载udf函数！", + "addUdf": "新建udf函数成功！", + "addFolder": "新建文件夹成功！", + "shareFunction": "函数已共享成功", + "updateShareUser": "修改共享用户成功！", + "functionExpired": "函数已设置为过期，请到共享函数中查看！" + }, + "confirm": { + "title": "提示", + "content": "

此操作将会对已共享的用户造成影响，是否继续？

" + } + }, + "hdfs": { + "contextMenu": { + "openToTab": "打开到侧边", + "copyPath": "复制路径", + "addCatalog": "新建目录", + "importToHive": "导入至Hive", + "exportToWorkSpace": "导出至工作空间" + }, + "dialogTitle": { + "exportToWorkSpace": "导出至工作空间" + }, + "warning": { + "noRootPath": "未请求到HDFS根目录，请刷新重试！" + } + }, + "container": { + "title": { + "editorName": "编辑器" + }, + "closeDropDown": { + "others": "关闭其他", + "all": "关闭全部", + "left": "关闭左边", + "right": "关闭右边" + }, + "closeHint": "关闭提示", + "footer": { + "cancel": "取消", + "close": "直接关闭", + "save": "保存", + "saveAs": "另存为" + }, + "tips": "什么是Scriptis？\nScriptis是微众银行微数域(WeDataSphere)打造的一站式交互式数据探索分析工具，以任意桥(Linkis)做为内核，提供多种计算存储引擎(如Spark、Hive、TiSpark等)、Hive数据库管理功能、资源(如Yarn资源、服务器资源)管理、应用管理和各种用户资源(如UDF、变量等)管理的能力。\nScriptis？\n1. 选中工作空间的目录，创建文件夹；\n2. 右键某个文件夹 =>新建脚本；\n3. 选择脚本类型，如：SQL、Pyspark、HQL等；\n4. 书写脚本，点击执行，生成结果集。", + "warning": { + "noDBInfo": "未获取到数据库表信息，脚本联想词功能可能存在异常！可刷新重试！", + "noUDF": "未获取到UDF和方法函数信息，脚本联想词功能可能存在异常！可刷新重试！", + "noSelectedScript": "未选中脚本，请选择一个脚本再试！" + }, + "notice": { + "developerWarning": { + "title": "警告", + "desc": "开发者警告：调用add接口必须传入id和filename！" + }, + "unSupport": { + "title": "警告", + "desc": "很抱歉，系统暂不支持打开该格式文件！" + } + }, + "removeWork": { + "normal": "该脚本已发生改变，是否保存？", + "readOnly": "该只读脚本已发生改变，是否另存至工作空间？", + "temporary": "该临时脚本已发生改变，是否另存为文件？" + } + }, + "editorDetail": { + "navBar": { + "undo": "撤销", + "redo": "恢复", + "play": "运行", + "stop": "停止", + "save": "保存", + "config": "配置" + }, + "warning": { + "running": "脚本运行中，请稍候！", + "invalidArgs": "请检查配置参数输入是否正确！", + "emptyCode": "无法执行空代码！" + } + }, + "history": { + "columns": { + "taskID": { + "title": "任务ID" + }, + "runningTime": { + "title": "执行时间" + }, + "createDate": { + "title": "启动时间" + }, + "status": { + "title": "状态", + "status": { + "succeed": "成功", + "running": "运行", + "timeout": "超时", + "inited": "排队中", + "scheduled": "资源申请中", + "failed": "失败", + "cancel": "取消" + } + }, + "data": { + "title": "代码" + }, + "failedReason": { + "title": "关键信息" + }, + "control": { + "title": "操作", + "view": "查看", + "download": "日志下载" + } + }, + "success": { + "open": "打开成功", + "download": "下载成功，请到本地的download文件夹查看！" + } + }, + "saveAs": { + "header": "另存为脚本", + "formItems": { + "fileName": { + "label": "文件名" + }, + "selectedPath": { + "label": "选择路径" + } + }, + "rules": { + "fileName": { + "required": "请输入文件名", + "lengthLimit": "长度在1到200个字符", + "letterTypeLimit": "仅支持中文、大小写字母、数字和下划线" + }, + "selectedPath": { + "required": "请选择一个目录" + } + }, + "warning": { + "invalid": "请查看未通过验证项！" + }, + "success": { + "saveScript": "另存脚本成功，请手动刷新目录树！" + } + }, + "tabs": { + "progress": "进度", + "result": "运行结果", + "log": "运行日志", + "history": "历史", + "diagnosis": "智能诊断" + }, + "notice": { + "unsave": { + "title": "提醒", + "desc": "您的临时脚本已修改，请注意保存！" + }, + "sendStart": { + "title": "运行提示", + "render": "开始执行脚本" + }, + "querySuccess": { + "title": "执行结果提示", + "render": "脚本执行成功，共耗时" + }, + "notice": { + "title": "执行结果提示" + }, + "kill": { + "title": "运行提示", + "desc": "已经停止执行脚本" + } + }, + "confirmModal": { + "title": "另存为", + "content": "

该临时脚本已发生改变！

是否另存？

", + "okText": "另存为", + "cancelText": "取消" + }, + "setting": { + "header": "配置", + "customVariable": { + "title": "自定义参数" + }, + "envVariable": { + "title": "环境变量" + }, + "runTimeArgs": { + "settingTitle": "命令行参数", + "placeholder": "请使用空格分开多个参数" + } + }, + "apiPublish": { + "title": "发布为数据API", + "button": { + "addApi": "新建数据API", + "updateApi": "更新数据API" + }, + "addApiModal": { + "modalTitle": "数据服务API新增", + "contentTitle": "基本信息补充", + "paramConfirmTitle": "参数确认", + "apiName": "API 名称", + "apiPath": "API Path", + "protocol": "协议", + "requestType": "请求方式", + "tag": "标签", + "visible": "可见范围", + "describe": "描述", + "cancel": "取消", + "nextStep": "下一步", + "approvalName": "审批单名称", + "applyUser": "申请用户", + "proxyUser": "代理执行用户", + "back": "上一步", + "confirm": "确认", + "comment": "备注" + }, + "updateApiModal": { + "modalTitle": "数据服务API更新", + "apiVersionUpgrade": "生成新版本", + "paramConfirm": "请确认参数" + }, + "paramTable": { + "paramName": "参数名称", + "paramType": "参数类型", + "require": { + "title": "是否必填", + "yes": "是", + "no": "否", + "hide": "隐藏" + }, + "defaultValue": "默认值", + "displayName": "展示名", + "describe": "描述" + }, + "rule": { + "nameRule": "请填写API名称", + "approvalName": "请填写名称", + "pathRule": "请填写API路径", + "pathRegRule": "API路径不合法", + "requestTypeRule": "请选择请求方式", + "protocolRule": "请选择协议", + "valueRule": "必须以字母开头，且只支持字母、数字、下划线！", + "contentLengthLimit": "长度不能大于150", + "contentLengthLimit50": "长度不能大于50", + "contentLengthLimitTwo": "长度不能大于200", + "pathRepeat": "API路径已存在", + "nameRepeat": "API名称已存在", + "chineseCheck": "只支持中文", + "chineseName": "API中文名称" + }, + "visible": { + "workspace": "工作空间", + "personal": "个人", + "public": "公开", + "grantView": "授权可见" + }, + "notice": { + "publishSuccess": "发布成功" + }, + "tagPlaceholder": "标签，按 enter 创建" + } + } + } +} diff --git a/web/src/js/module/dateReport/index.js b/web/src/apps/scriptis/module/dateReport/index.js old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/module/dateReport/index.js rename to web/src/apps/scriptis/module/dateReport/index.js diff --git a/web/src/apps/scriptis/module/dateReport/index.vue b/web/src/apps/scriptis/module/dateReport/index.vue new file mode 100755 index 000000000..f42c3f9aa --- /dev/null +++ b/web/src/apps/scriptis/module/dateReport/index.vue @@ -0,0 +1,27 @@ + + + diff --git a/web/src/apps/scriptis/module/workbench/dbDetails/components/basic.vue b/web/src/apps/scriptis/module/workbench/dbDetails/components/basic.vue new file mode 100755 index 000000000..2d214671d --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/dbDetails/components/basic.vue @@ -0,0 +1,112 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/dbDetails/components/tableList.vue b/web/src/apps/scriptis/module/workbench/dbDetails/components/tableList.vue new file mode 100755 index 000000000..8fde2bb0e --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/dbDetails/components/tableList.vue @@ -0,0 +1,224 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/dbDetails/index.vue b/web/src/apps/scriptis/module/workbench/dbDetails/index.vue new file mode 100755 index 000000000..196c71916 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/dbDetails/index.vue @@ -0,0 +1,120 @@ + + + + diff --git a/web/src/js/module/workbench/image/bg-img.png b/web/src/apps/scriptis/module/workbench/image/bg-img.png old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/module/workbench/image/bg-img.png rename to web/src/apps/scriptis/module/workbench/image/bg-img.png diff --git a/web/src/js/module/workbench/image/hive.png b/web/src/apps/scriptis/module/workbench/image/hive.png old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/module/workbench/image/hive.png rename to web/src/apps/scriptis/module/workbench/image/hive.png diff --git a/web/src/js/module/workbench/image/python.png b/web/src/apps/scriptis/module/workbench/image/python.png old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/module/workbench/image/python.png rename to web/src/apps/scriptis/module/workbench/image/python.png diff --git a/web/src/js/module/workbench/image/scala.png b/web/src/apps/scriptis/module/workbench/image/scala.png old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/module/workbench/image/scala.png rename to web/src/apps/scriptis/module/workbench/image/scala.png diff --git a/web/src/js/module/workbench/image/spark.png b/web/src/apps/scriptis/module/workbench/image/spark.png old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/module/workbench/image/spark.png rename to web/src/apps/scriptis/module/workbench/image/spark.png diff --git a/web/src/apps/scriptis/module/workbench/index.js b/web/src/apps/scriptis/module/workbench/index.js new file mode 100755 index 000000000..5e51102c7 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/index.js @@ -0,0 +1,50 @@ +/* + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +export default { + name: 'Workbench', + // 规范模块监测什么事件 + events: ['Workbench:add', 'Workbench:deleteDirOrFile', 'Workbench:save', 'Workbench:socket', 'Workbench:openFile', 'Workbench:run', 'Workbench:pasteInEditor', 'Workbench:saveAs', 'Workbench:updateTab', 'Workbench:setResultCache', 'Workbench:setResult', 'Workbench:insertValue', 'Workbench:checkExist', 'Workbench:getWorksLangList', 'Workbench:setEditorPanelSize', 'Workbench:setTabPanelSize', 'Workbench:updateFlowsTab', 'Workbench:resetScriptData', 'Workbench:removeWork', 'Workbench:updateFlowsNodeName', 'Workbench:isOpenTab', 'Workbench:removeTab'], + // 规范模块能够触发什么事件 + dispatchs: { + Workbench: ['save', 'saveAs', 'run', 'add', 'setResultCache', 'setResult', 'insertValue', 'openFile', 'checkExist', 'setEditorPanelSize', 'setTabPanelSize', 'resetScriptData', 'removeWork', 'repeatWork', 'removeTab'], + IndexedDB: [ + 'appendLog', 'clearLog', 'getLog', 'changeLogKey', + 'updateHistory', 'appendHistory', 'getHistory', 'clearHistory', 'changeHistoryKey', + 'updateResult', 'appendResult', 'getResult', 'clearResult', 'changResultKey', + 'updateProgress', 'clearProgress', 'getProgress', 'changProgressKey', + 'getTabs', 'recordTab', 'toggleTab', 'removeTab', 'changeTabKey', + 'getGlobalCache', 'setGlobalCache', 'updateGlobalCache', 'removeGlobalCache', 'getTree', 'appendTree' + ], + WebSocket: ['init', 'send'], + WorkSidebar: ['setHighLight', 'showTree', 'revealInSideBar'], + HdfsSidebar: ['setHighLight', 'showTree'], + HiveSidebar: ['getAllDbsAndTables', 'deletedAndRefresh', 'showHive', 'getAllowMap'], + fnSidebar: ['getAllLoadedFunction'], + Footer: ['updateRunningJob'], + GlobalValiable: ['getGlobalVariable'], + IDE: ['saveNode'], + }, + data: { + API_PATH: process.env.VUE_APP_MN_CONFIG_PREFIX || `http://${window.location.host}/api/rest_j/v1/`, + BI_API_PATH: 'dws/vg#', + }, + component: () => + import('./container.vue'), + config: { + }, +}; diff --git a/web/src/apps/scriptis/module/workbench/modal.js b/web/src/apps/scriptis/module/workbench/modal.js new file mode 100755 index 000000000..f1486a7f5 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/modal.js @@ -0,0 +1,163 @@ +/* + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +let logoList = [ + { rule: /\.(bi)$/i, logo: 'fi-bi' }, + { rule: /\.(sql)$/i, logo: 'fi-spark' }, + { rule: /\.(hql)$/i, logo: 'fi-hive' }, + { rule: /\.(fql)$/i, logo: 'fi-flink-sql' }, + { rule: /\.(out)$/i, logo: 'fi-storage' }, + { rule: /\.scala$/i, logo: 'fi-scala' }, + { rule: /\.jdbc$/i, logo: 'fi-jdbc' }, + { rule: /\.python$/i, logo: 'fi-python' }, + { rule: /\.py$/i, logo: 'fi-spark-python' }, + { rule: /\.r$/i, logo: 'fi-r' }, + { rule: /\.qmlsql$/i, logo: 'fi-spark' }, + { rule: /\.qmlpy$/i, logo: 'fi-python' }, + { rule: /\.txt$/i, logo: 'fi-txt' }, + { rule: /\.log$/i, logo: 'fi-log' }, +]; + +/** + * 基础 + */ +export class Work { + /** + * 构造器 + * @param {*} option + */ + constructor(option) { + // 唯一标识 + this.id = option.id; + // 执行id + this.execID = null; + // 历史脚本才用得到，只/api/jobhistory/${id}/get请求使用 + this.taskID = option.taskID; + // 文件名 + this.filename = option.filename; + // 文件路径 + this.filepath = option.filepath; + // 数据开发模块会显示文件列表 + this.fileList = option.fileList || []; + // 是否保存 + this.unsave = option.unsave || false; + // 是否另存 + this.saveAs = option.saveAs || false; + // 是否要读取缓存数据，一般用于自动运行的时候 + this.noLoadCache = option.noLoadCache || false; + // 类型 + this.type = option.type || 'workspaceScript'; // workspaceScript hdfsScript historyScript workflow backgroundScript + // 数据 + this.data = option.data || null; + // 脚本内容和参数（test） + this.code = option.code; + this.params = option.params; + // 用于记录临时脚本或者HDFS脚本另存后保存内容和参数用 + this.ismodifyByOldTab = option.ismodifyByOldTab || false; + this.owner = option.owner; + this.specialSetting = option.specialSetting; + this.nodeName = option.nodeName || null; // 在工作流操作时记录名称 + } + + /** + * 根据文件后缀判断logo + */ + get logo() { + let logos = logoList.filter((item) => { + return item.rule.test(this.filename); + }); + if (logos.length > 0) { + return logos[0].logo; + } else { + return 'javascript: void 0'; + } + } +} + +/** + * 脚本任务 + */ +export class Script { + /** + * 构造器 + * @param {*} option + */ + constructor(option) { + this.id = option.id; + this.fileName = option.fileName; + this.filepath = option.filepath; + // 脚本内容 + this.data = option.data || option.code || ''; + // 脚本原内容 + this.oldData = option.data || ''; + // 日志 + this.log = option.log || {}; + this.logLine = option.logLine || 1; + // 历史 + this.history = []; + // 进度 + this.progress = { + current: null, + progressInfo: [], + waitingSize: null, + costTime: null, + }; + // 步骤 + this.steps = []; + // 智能诊断 + this.diagnosis = null; + // 运行结果 + this.result = null; + // 记录结果集的存储路径 + this.resultList = null; + // 参数 + this.params = option.params || { + variable: [], + configuration: { + special: {}, + runtime: { + args: '', + env: [], + }, + startup: {}, + }, + }; + + // editor组件的language + this.lang = option.lang; + // 是否可执行 + this.executable = option.executable || false; + // 是否可配置 + this.configurable = option.configurable || true; + // 后台使用哪种BDP服务 + this.application = option.application; + // 后台运行的服务类型 + this.runType = option.runType; + // 后缀 + this.ext = option.ext; + // 否为系统支持的脚本类型 + this.scriptType = option.scriptType; + // 是否可读 + this.readOnly = option.readOnly || false; + // 是否正在执行 + this.running = false; + // 当前的运行状态 + this.status = option.status ? option.status : 'Inited'; + // script视图状态 + this.scriptViewState = {}; + } +} diff --git a/web/src/js/module/workbench/script/backgroundScript.vue b/web/src/apps/scriptis/module/workbench/script/backgroundScript.vue old mode 100644 new mode 100755 similarity index 88% rename from web/src/js/module/workbench/script/backgroundScript.vue rename to web/src/apps/scriptis/module/workbench/script/backgroundScript.vue index 5c3ab5b6e..a8bb5bb88 --- a/web/src/js/module/workbench/script/backgroundScript.vue +++ b/web/src/apps/scriptis/module/workbench/script/backgroundScript.vue @@ -2,14 +2,14 @@
+ + diff --git a/web/src/apps/scriptis/module/workbench/script/history.vue b/web/src/apps/scriptis/module/workbench/script/history.vue new file mode 100755 index 000000000..7b3679688 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/script/history.vue @@ -0,0 +1,301 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/script/intelligentDiagnosis.vue b/web/src/apps/scriptis/module/workbench/script/intelligentDiagnosis.vue new file mode 100755 index 000000000..ac2c83363 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/script/intelligentDiagnosis.vue @@ -0,0 +1,55 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/script/mixin.js b/web/src/apps/scriptis/module/workbench/script/mixin.js new file mode 100755 index 000000000..996e99bf7 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/script/mixin.js @@ -0,0 +1,44 @@ +/* + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +import { throttle } from 'lodash'; +import elementResizeEvent from '@/common/helper/elementResizeEvent'; +export default { + data() { + return { + tableHeight: 100, + }; + }, + mounted() { + this.initEvents(); + }, + beforeDestroy: function() { + elementResizeEvent.unbind(this.$el); + }, + methods: { + initEvents() { + let thro = throttle(() => { + this.resize(); + }); + elementResizeEvent.bind(this.$el, thro); + }, + resize() { + this.tableHeight = this.$el.clientHeight - 40; + }, + }, +} +; diff --git a/web/src/apps/scriptis/module/workbench/script/saveAs.vue b/web/src/apps/scriptis/module/workbench/script/saveAs.vue new file mode 100755 index 000000000..41c3cb5f5 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/script/saveAs.vue @@ -0,0 +1,200 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/script/script.vue b/web/src/apps/scriptis/module/workbench/script/script.vue new file mode 100755 index 000000000..8aa8c8fa6 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/script/script.vue @@ -0,0 +1,1543 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/script/setting.vue b/web/src/apps/scriptis/module/workbench/script/setting.vue new file mode 100755 index 000000000..a66b43aae --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/script/setting.vue @@ -0,0 +1,102 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/setting/customVariable.vue b/web/src/apps/scriptis/module/workbench/setting/customVariable.vue new file mode 100755 index 000000000..640f8feab --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/setting/customVariable.vue @@ -0,0 +1,24 @@ + + diff --git a/web/src/apps/scriptis/module/workbench/setting/envVariable.vue b/web/src/apps/scriptis/module/workbench/setting/envVariable.vue new file mode 100755 index 000000000..46fab494c --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/setting/envVariable.vue @@ -0,0 +1,26 @@ + + diff --git a/web/src/apps/scriptis/module/workbench/setting/runTimeArgs.vue b/web/src/apps/scriptis/module/workbench/setting/runTimeArgs.vue new file mode 100755 index 000000000..334829aa4 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/setting/runTimeArgs.vue @@ -0,0 +1,32 @@ + + diff --git a/web/src/apps/scriptis/module/workbench/tableDetails/components/basic.vue b/web/src/apps/scriptis/module/workbench/tableDetails/components/basic.vue new file mode 100755 index 000000000..cfa9be1df --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/tableDetails/components/basic.vue @@ -0,0 +1,162 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/tableDetails/components/field.vue b/web/src/apps/scriptis/module/workbench/tableDetails/components/field.vue new file mode 100755 index 000000000..12594b29a --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/tableDetails/components/field.vue @@ -0,0 +1,169 @@ + + + \ No newline at end of file diff --git a/web/src/apps/scriptis/module/workbench/tableDetails/components/statistics.vue b/web/src/apps/scriptis/module/workbench/tableDetails/components/statistics.vue new file mode 100755 index 000000000..dba19c85d --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/tableDetails/components/statistics.vue @@ -0,0 +1,235 @@ + + + diff --git a/web/src/apps/scriptis/module/workbench/tableDetails/index.vue b/web/src/apps/scriptis/module/workbench/tableDetails/index.vue new file mode 100755 index 000000000..4459863b9 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/tableDetails/index.vue @@ -0,0 +1,185 @@ + + + diff --git a/web/src/apps/scriptis/module/workbench/tableDetails/utils.js b/web/src/apps/scriptis/module/workbench/tableDetails/utils.js new file mode 100755 index 000000000..7e34cd421 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/tableDetails/utils.js @@ -0,0 +1,93 @@ +/* + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +import moment from 'moment'; + +const convertList = { + lifecycle: [{ + value: 0, + label: '永久', + }, { + value: 1, + label: '当天有效', + }, { + value: 2, + label: '一周有效', + }, { + value: 3, + label: '一月有效', + }, { + value: 4, + label: '半年有效', + }], + modelLevel: [{ + value: 0, + label: 'ODS(原始数据层)', + }, { + value: 1, + label: 'DWD(明细数据层)', + }, { + value: 2, + label: 'DWS(汇总数据层)', + }, { + value: 3, + label: 'ADS(应用数据层)', + }], + useWay: [{ + value: 0, + label: '一次写多次读', + }, { + value: 1, + label: '增删改查', + }, { + value: 2, + label: '多次覆盖写', + }, { + value: 3, + label: '一次写偶尔读', + }], +}; + +/** + * 格式化值 + * @param {*} item + * @param {*} field + * @return {*} return + */ +function formatValue(item, field) { + const value = item[field.key]; + let formatted = value; + switch (field.type) { + case 'boolean': + formatted = value ? '是' : '否'; + break; + case 'timestramp': + formatted = value == '0' || !value ? 0 : moment.unix(value).format('YYYY-MM-DD HH:mm:ss'); + break; + case 'convert': + if (!item[field.key] && item[field.key] !== 0) { + return value; + } + formatted = convertList[field.key][item[field.key]].label; + break; + } + return formatted; +} + +export default { + formatValue, +}; diff --git a/web/src/apps/scriptis/module/workbench/title.vue b/web/src/apps/scriptis/module/workbench/title.vue new file mode 100755 index 000000000..49f71ccf7 --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/title.vue @@ -0,0 +1,117 @@ + + + + diff --git a/web/src/apps/scriptis/module/workbench/visualAnalysis/visualAnalysis.vue b/web/src/apps/scriptis/module/workbench/visualAnalysis/visualAnalysis.vue new file mode 100755 index 000000000..e91d330de --- /dev/null +++ b/web/src/apps/scriptis/module/workbench/visualAnalysis/visualAnalysis.vue @@ -0,0 +1,128 @@ + + + diff --git a/web/src/apps/scriptis/router.js b/web/src/apps/scriptis/router.js new file mode 100755 index 000000000..448bdf6a6 --- /dev/null +++ b/web/src/apps/scriptis/router.js @@ -0,0 +1,41 @@ +/* + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +export const subAppRoutes = { + path: '', + name: 'layout', + component: () => import('./view/layout.vue'), + redirect: '/home', + meta: { + title: 'DataSphere Studio', + publicPage: true, // 权限公开 + }, + children: [] +} + +export default [ + { + path: 'home', + name: 'Home', + meta: { + title: 'Scriptis', + keepAlive: false, // 缓存导致页面有多个编辑器，广播事件会触发报错 + publicPage: true, // 权限公开 + }, + component: () => + import('./view/home/index.vue'), + } +] diff --git a/web/src/js/service/db/globalcache.js b/web/src/apps/scriptis/service/db/globalcache.js old mode 100644 new mode 100755 similarity index 79% rename from web/src/js/service/db/globalcache.js rename to web/src/apps/scriptis/service/db/globalcache.js index 9cce9a03e..46725dd57 --- a/web/src/js/service/db/globalcache.js +++ b/web/src/apps/scriptis/service/db/globalcache.js @@ -15,8 +15,8 @@ * */ -import _ from 'lodash'; -import { Basic } from './index.js'; +import { isNil, remove } from 'lodash'; +import { Basic } from '@/common/service/db/index.js'; /** * @class Globalcache * @extends {Basic} @@ -28,7 +28,7 @@ class Globalcache extends Basic { * @param {*} db * @memberof Globalcache */ - constructor(table, db) { + constructor(table) { super(table); } @@ -39,7 +39,7 @@ class Globalcache extends Basic { async setCache(args) { let cache = await this.getCache(args.key); let cacheToUpdate = args; - if (!_.isNil(cache)) { + if (!isNil(cache)) { cacheToUpdate = { _id: cache._id, ...cacheToUpdate }; } return this.add(cacheToUpdate); @@ -50,7 +50,7 @@ class Globalcache extends Basic { * @return {*} */ async getCache(key) { - let caches = await this.get(key); + let caches = await this.get(key) || []; return caches[0]; } @@ -61,9 +61,9 @@ class Globalcache extends Basic { async removeCache(args) { let cache = await this.getCache(args.id); let tabList = []; - if (!_.isNil(cache)) { + if (!isNil(cache)) { tabList = cache.tabList; - _.remove(tabList, (n) => n === args.tabId); + remove(tabList, (n) => n === args.tabId); } return this.update(args.id, { key: args.id, tabList: tabList }); } @@ -77,7 +77,7 @@ class Globalcache extends Basic { let cache = await this.getCache(args.id); const id = args.work.id; let tabList = []; - if (!_.isNil(cache)) { + if (!isNil(cache)) { tabList = cache.tabList; if (cache.tabList.indexOf(id) === -1) { tabList.push(id); @@ -86,11 +86,14 @@ class Globalcache extends Basic { tabList.push(id); } return this.update(args.id, { key: args.id, tabList: tabList }); - } else if (args.fnList) { - return this.update(args.id, { key: args.id, fnList: args.fnList }); - } else if (args.variableList) { - return this.update(args.id, { key: args.id, variableList: args.variableList }); } + if (args.fnList) { + this.update(args.id, { key: args.id, fnList: args.fnList }); + } + if (args.variableList) { + this.update(args.id, { key: args.id, variableList: args.variableList }); + } + return; } } const globalcache = new Globalcache('globalCache'); diff --git a/web/src/js/service/db/history.js b/web/src/apps/scriptis/service/db/history.js old mode 100644 new mode 100755 similarity index 94% rename from web/src/js/service/db/history.js rename to web/src/apps/scriptis/service/db/history.js index 47bf66d9e..09da6d759 --- a/web/src/js/service/db/history.js +++ b/web/src/apps/scriptis/service/db/history.js @@ -15,7 +15,7 @@ * */ -import { Basic } from './index.js'; +import { Basic } from '@/common/service/db/index.js'; /** * @class History * @extends {Basic} diff --git a/web/src/js/service/db/log.js b/web/src/apps/scriptis/service/db/log.js old mode 100644 new mode 100755 similarity index 94% rename from web/src/js/service/db/log.js rename to web/src/apps/scriptis/service/db/log.js index 5e9992ad9..abb67137b --- a/web/src/js/service/db/log.js +++ b/web/src/apps/scriptis/service/db/log.js @@ -15,7 +15,7 @@ * */ -import { Basic } from './index.js'; +import { Basic } from '@/common/service/db/index.js'; /** * @class Log * @extends {Basic} diff --git a/web/src/js/service/db/progress.js b/web/src/apps/scriptis/service/db/progress.js old mode 100644 new mode 100755 similarity index 94% rename from web/src/js/service/db/progress.js rename to web/src/apps/scriptis/service/db/progress.js index bc2c99d8f..7cb29e8f1 --- a/web/src/js/service/db/progress.js +++ b/web/src/apps/scriptis/service/db/progress.js @@ -15,7 +15,7 @@ * */ -import { Basic } from './index.js'; +import { Basic } from '@/common/service/db/index.js'; /** * @class progress * @extends {Basic} diff --git a/web/src/js/service/db/result.js b/web/src/apps/scriptis/service/db/result.js old mode 100644 new mode 100755 similarity index 94% rename from web/src/js/service/db/result.js rename to web/src/apps/scriptis/service/db/result.js index 7f0dc7eae..6558f72ac --- a/web/src/js/service/db/result.js +++ b/web/src/apps/scriptis/service/db/result.js @@ -15,7 +15,7 @@ * */ -import { Basic } from './index.js'; +import { Basic } from '@/common/service/db/index.js'; /** * @class Result * @extends {Basic} diff --git a/web/src/js/service/db/tab.js b/web/src/apps/scriptis/service/db/tab.js old mode 100644 new mode 100755 similarity index 94% rename from web/src/js/service/db/tab.js rename to web/src/apps/scriptis/service/db/tab.js index 4b368ca00..aca5dc17a --- a/web/src/js/service/db/tab.js +++ b/web/src/apps/scriptis/service/db/tab.js @@ -15,7 +15,7 @@ * */ -import { Basic } from './index.js'; +import { Basic } from '@/common/service/db/index.js'; /** * @class Tab * @extends {Basic} diff --git a/web/src/apps/scriptis/service/db/tree.js b/web/src/apps/scriptis/service/db/tree.js new file mode 100755 index 000000000..7907ea50f --- /dev/null +++ b/web/src/apps/scriptis/service/db/tree.js @@ -0,0 +1,35 @@ +/* + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +import { Basic } from '@/common/service/db/index.js'; +/** + * @class Tab + * @extends {Basic} + */ +class Tree extends Basic { + /** + *Creates an instance of Tab. + * @param {*} table + * @memberof Tree + */ + constructor(table) { + super(table); + } +} +const tree = new Tree('tree'); + +export default tree; diff --git a/web/src/apps/scriptis/view/home/index.vue b/web/src/apps/scriptis/view/home/index.vue new file mode 100755 index 000000000..9b246b509 --- /dev/null +++ b/web/src/apps/scriptis/view/home/index.vue @@ -0,0 +1,123 @@ + + + diff --git a/web/src/apps/scriptis/view/layout.vue b/web/src/apps/scriptis/view/layout.vue new file mode 100755 index 000000000..8958c366c --- /dev/null +++ b/web/src/apps/scriptis/view/layout.vue @@ -0,0 +1,20 @@ + + diff --git a/web/src/apps/workflows/assets/styles/process.scss b/web/src/apps/workflows/assets/styles/process.scss new file mode 100755 index 000000000..4bd62873d --- /dev/null +++ b/web/src/apps/workflows/assets/styles/process.scss @@ -0,0 +1,154 @@ +/*! + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +@import '@/common/style/variables.scss'; +.page-process{ + height: 100%; + width: 100%; + background-color: $background-color-base; + position: relative; + .bread-crumb{ + margin: 5px; + } + .process-readonly-tip-card { + color: #FFF; + font-size: $font-size-small; + position: absolute; + font-weight: 700; + z-index: 9999; + top: 0; + right: 1rem; + background-color: $error-color; + .tipClose { + position: absolute; + top: 5px; + right: 5px; + color: #080808; + } + } +} + +.process-tabs { + position: $relative; + height: calc(100%); + overflow: hidden; + &.no-tab{ + .process-tab{ + display: none; + background: $body-background; + } + .process-container{ + height: 100%; + } + } + .process-tab { + display: flex; + flex-direction: row; + flex-wrap: nowrap; + justify-content: flex-start; + align-items: flex-start; + height: 32px; + padding: 0 20px; + box-sizing: border-box; + border-top: $border-width-base $border-style-base $border-color-split; + background: $body-background; + .process-tab-item { + position: $relative; + height: 30px; + line-height: 30px; + padding: 0 20px; + padding-left: 25px; + border-right: $border-width-base $border-style-base $border-color-split; + background: $background-color-base; + color: $title-color; + cursor: pointer; + min-width: 100px; + max-width: 200px; + overflow: hidden; + text-align: center; + &:first-child{ + border-left: $border-width-base $border-style-base $border-color-split; + } + &.active { + margin-top: -1px; + &:before{ + content: ""; + position: absolute; + top: -1px; + left: 0; + right: 0; + height: 3px; + background: $primary-color; + } + height: 32px; + line-height: 31px; + background: $body-background; + color: $primary-color; + .tab-icon { + top: 9px; + } + } + .process-tab-name{ + width: 100%; + overflow: hidden; + white-space: nowrap; + text-overflow: ellipsis; + padding-right: 5px; + padding-left: 5px; + font-size: $font-size-large; + } + .process-tab-unsave-icon { + position: absolute; + top: 50%; + right: 10px; + transform: translateY(-50%); + } + .ivu-icon{ + position: absolute; + right: 10px; + top: 10px; + } + .tab-icon { + position: absolute; + left: 7px; + top: 8px; + width: 15px; + height: 15px; + vertical-align: middle; + } + // .flow { + // width: 12px; + // height: 12px; + // } + .hivesql { + width: 16px; + height: 16px; + } + // .display { + // width: 13px; + // height: 13px; + // } + .dashboard { + width: 14px; + height: 14px; + } + } + } + .process-container { + height: calc(100% - 32px); + } +} diff --git a/web/src/apps/workflows/assets/styles/workflow.scss b/web/src/apps/workflows/assets/styles/workflow.scss new file mode 100755 index 000000000..38c186c75 --- /dev/null +++ b/web/src/apps/workflows/assets/styles/workflow.scss @@ -0,0 +1,88 @@ +/*! + * Copyright 2019 WeBank + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +@import '@/common/style/variables.scss'; +.workflow-wrap { + width: 100%; + height: 100%; + background: $workspace-background; + .workflowListContainer { + display: flex; + margin: 0 $padding-25; + min-height: 685px; + padding: $padding-25; + border: $border-width-base $border-style-base $border-color-base; + background: $body-background; + .workflowListLeft { + flex: 1; + .tabs-content { + /deep/ .ivu-tabs-tab { + font-size: $font-size-large; + } + } + } + } + .workflow-tabs { + width: 100%; + height: 32px; + margin: 4px 0; + white-space: nowrap; + line-height: $line-height-base; + font-size: $font-size-base; + position: relative; + color: $text-color; + border-bottom: $border-width-base $border-style-base $border-color-base; + padding: 0 5px; + .workflow-tabs-item { + margin: 0; + height: 31px; + padding: 5px 16px 4px; + border-bottom: $border-width-base $border-style-base $border-color-base; + border-radius: 4px 4px 0 0; + background: #f8f8f9; + display: inline-block; + cursor: pointer; + position: relative; + &.active { + height: 32px; + padding-bottom: 5px; + background: #fff; + transform: translateZ(0); + border: 1px solid #dcdee2; + border-bottom:1px solid #fff; + color: #2d8cf0; + } + .workflow-tabs-name { + display: inline-block; + } + .workflow-tabs-close { + width: 22px; + margin-right: -6px; + height: 22px; + font-size: 22px; + color: #999; + text-align: right; + vertical-align: middle; + overflow: hidden; + position: relative; + top: -1px; + transform-origin: 100% 50%; + transition: all .3s ease-in-out; + cursor: pointer; + } + } + } +} diff --git a/web/src/js/component/vue-process/actionView.vue b/web/src/apps/workflows/components/vue-process/actionView.vue old mode 100644 new mode 100755 similarity index 100% rename from web/src/js/component/vue-process/actionView.vue rename to web/src/apps/workflows/components/vue-process/actionView.vue diff --git a/web/src/js/component/vue-process/baseInfo.vue b/web/src/apps/workflows/components/vue-process/baseInfo.vue old mode 100644 new mode 100755 similarity index 77% rename from web/src/js/component/vue-process/baseInfo.vue rename to web/src/apps/workflows/components/vue-process/baseInfo.vue index 37909cf7d..317961fd2 --- a/web/src/js/component/vue-process/baseInfo.vue +++ b/web/src/apps/workflows/components/vue-process/baseInfo.vue @@ -1,24 +1,24 @@