[checkpoint_email] Initial release of the Check Point Harmony Email &…

… Collaboration (elastic#11472) Create New integration package checkpoint_email. Added event data stream. Added data collection logic for the data stream. Added the ingest pipeline for the data stream. Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files. Added dashboard and visualizations. Added pipeline test for the data stream. Added system test cases for the data stream.
srilumpa · Nov 8, 2024 · 9b5a6b4 · 9b5a6b4
1 parent c3c5e7b
commit 9b5a6b4
Show file tree

Hide file tree

Showing 26 changed files with 3,793 additions and 0 deletions.
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -129,6 +129,7 @@
 /packages/cel @elastic/security-service-integrations
 /packages/ceph @elastic/obs-infraobs-integrations
 /packages/checkpoint @elastic/sec-deployment-and-devices
+/packages/checkpoint_email @elastic/security-service-integrations
 /packages/checkpoint_harmony_endpoint @elastic/security-service-integrations
 /packages/cisa_kevs @elastic/security-service-integrations
 /packages/cisco_aironet @elastic/sec-deployment-and-devices

diff --git a/packages/checkpoint_email/_dev/build/build.yml b/packages/checkpoint_email/_dev/build/build.yml
@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]
diff --git a/packages/checkpoint_email/_dev/build/docs/README.md b/packages/checkpoint_email/_dev/build/docs/README.md
@@ -0,0 +1,72 @@
+# Check Point Harmony Email & Collaboration
+
+Check Point's [Harmony Email & Collaboration](https://www.checkpoint.com/harmony/email-security/) monitors traffic across email platforms (Office 365, Gmail), file sharing services (OneDrive, SharePoint, Google Drive, Dropbox, Box, and Citrix ShareFile), and messaging applications (Teams and Slack). It scans emails, files, and messages for malware, DLP, and phishing indicators, and intercepts & quarantines potentially malicious emails before they are delivered.
+
+The Check Point Harmony Email & Collaboration integration collects security event logs using REST API.
+
+## Data streams
+
+This integration collects the following logs:
+
+- **[Event](https://app.swaggerhub.com/apis-docs/Check-Point/harmony-email-collaboration-smart-api/1.50#/APIs/query_event_v1_0_event_query_post)** - Get security event logs.
+
+## Requirements
+
+Elastic Agent must be installed. For more details and installation instructions, please refer to the [Elastic Agent Installation Guide](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+
+### Installing and managing an Elastic Agent:
+
+There are several options for installing and managing Elastic Agent:
+
+### Install a Fleet-managed Elastic Agent (recommended):
+
+With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.
+
+### Install Elastic Agent in standalone mode (advanced users):
+
+With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.
+
+### Install Elastic Agent in a containerized environment:
+
+You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
+
+Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the [Elastic Agent Minimum Requirements](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#elastic-agent-installation-minimum-requirements).
+
+## Setup
+
+### To collect data from the Harmony Email & Collaboration Smart API:
+
+- In the Infinity Portal, go to Account Settings and click **API Keys**.
+- Click **New** > **New Account API key**.
+- In the **Create a New API Key** window, select **Email & Collaboration** as the service.
+- (Optional) In the **Expiration** field, select an expiration date and time for the API key. By default, the expiration date is three months after the creation date.
+- (Optional) In the **Description** field, enter a description for the API key.
+- Click **Create**.
+- Copy the **Client ID** and **Secret Key**.
+    - **Note**: You can always obtain the **Client ID** from the **API Keys** table, but you cannot retrieve the **Secret Key** after the **Create a New API Key** window is closed.
+- Click **Close**.
+
+For more details, see [Documentation](https://sc1.checkpoint.com/documents/Infinity_Portal/WebAdminGuides/EN/Infinity-Portal-Admin-Guide/Content/Topics-Infinity-Portal/API-Keys.htm?tocpath=Account%20Settings%7C_____7#API_Keys).
+
+### Enabling the integration in Elastic:
+
+1. In Kibana navigate to Management > Integrations.
+2. In "Search for integrations" top bar, search for `Check Point Harmony Email & Collaboration`.
+3. Select the "Check Point Harmony Email & Collaboration" integration from the search results.
+4. Select "Add Check Point Harmony Email & Collaboration" to add the integration.
+5. Add all the required integration configuration parameters, including the URL, Client ID, Client Secret, Interval, and Initial Interval, to enable data collection.
+6. Select "Save and continue" to save the integration.
+
+**Note**: The default URL is `https://cloudinfra-gw.portal.checkpoint.com`, but this may vary depending on your region. Please refer to the [Documentation](https://sc1.checkpoint.com/documents/Harmony_Email_and_Collaboration_API_Reference/Topics-HEC-Avanan-API-Reference-Guide/Overview/URLs-and-URL-Base.htm?tocpath=Executing%20API%20Calls%7C_____3) to find the correct URL for your region.
+
+## Logs reference
+
+### Event
+
+This is the `event` dataset.
+
+#### Example
+
+{{event "event"}}
+
+{{fields "event"}}
diff --git a/packages/checkpoint_email/_dev/deploy/docker/docker-compose.yml b/packages/checkpoint_email/_dev/deploy/docker/docker-compose.yml
@@ -0,0 +1,15 @@
+version: '2.3'
+services:
+  checkpoint_email:
+    image: docker.elastic.co/observability/stream:v0.15.0
+    hostname: checkpoint_email
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config.yml
diff --git a/packages/checkpoint_email/_dev/deploy/docker/files/config.yml b/packages/checkpoint_email/_dev/deploy/docker/files/config.yml
@@ -0,0 +1,203 @@
+rules:
+  - path: /auth/external
+    methods: ['POST']
+    request_headers:
+      Content-Type:
+        - "application/json"
+    request_body: '{"accessKey":"accessKey","clientId":"clientId"}'
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {
+            "success": true,
+            "data": {
+              "token": "XnbabcdefghijklmnopqrstuvwxyztYWHWXRy7---s",
+              "csrf": "32057d5e-abcd-1234-abcd-123456789087",
+              "expires": "Mon, 14 Oct 2030 07:20:18 GMT",
+              "expiresIn": 1800
+            }
+          }
+  - path: /app/hec-api/v1.0/event/query
+    methods: ['POST']
+    request_headers:
+      Content-Type:
+        - "application/json"
+      Authorization:
+        - "Bearer XnbabcdefghijklmnopqrstuvwxyztYWHWXRy7---s"
+      X-Av-Req-Id:
+        - .*
+    request_body: /^\{"requestData":\{"endDate":.*,"scrollId":"e9c8c0e01abcdefabcdef456789000cf","startDate":.*\}\}/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {
+            "responseEnvelope": {
+              "requestId": "{{ .request.headers.Get "X-Av-Req-Id" }}",
+              "responseCode": 200,
+              "responseText": "",
+              "additionalText": "",
+              "recordsNumber": 4,
+              "scrollId": null
+            },
+            "responseData": []
+          }
+  - path: /app/hec-api/v1.0/event/query
+    methods: ['POST']
+    request_headers:
+      Content-Type:
+        - "application/json"
+      Authorization:
+        - "Bearer XnbabcdefghijklmnopqrstuvwxyztYWHWXRy7---s"
+      X-Av-Req-Id:
+        - .*
+    request_body: /^\{"requestData":\{"endDate":.*,"scrollId":"e9c8c0e01234567890123456789000cf","startDate":.*\}\}/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {
+            "responseEnvelope": {
+              "requestId": "{{ .request.headers.Get "X-Av-Req-Id" }}",
+              "responseCode": 200,
+              "responseText": "",
+              "additionalText": "",
+              "recordsNumber": 4,
+              "scrollId": "e9c8c0e01abcdefabcdef456789000cf"
+            },
+            "responseData": [
+              {
+                "eventId": "a6d8674a04c30123456789e4d3ebd98",
+                "customerId": "exampletest",
+                "saas": "google_mail",
+                "entityId": "25e0c50123456789e351b0dafa6aafa6",
+                "state": "pending",
+                "type": "shadow_it",
+                "confidenceIndicator": "detected",
+                "eventCreated": "2024-10-14T07:02:11.229935+00:00",
+                "severity": "3",
+                "description": "Shadow IT - [email protected] is using google.com (Search Engine)",
+                "data": "#{\"entity_id\": \"a6d8674a04c30123456789e4d3ebd98\", \"entity_type\": \"google_mail_email\", \"label\": \"Shadow IT\"} - #{\"entity_id\": \"113012345678906535444\", \"entity_type\": \"google_user\", \"label\": \"[email protected]\"} is using #{\"entity_id\": \"google.com\", \"entity_type\": \"av_dns_info\", \"label\": \"google.com (Search Engine)\"}",
+                "additionalData": null,
+                "availableEventActions": null,
+                "actions": [],
+                "senderAddress": "[email protected]",
+                "entityLink": "https://in.portal.checkpoint.com/dashboard/email&collaboration/CGS1?route=cHJvZmlsZS9nsfhvbksdvnjhvdfVBsdbdfFbdbdBDBBdbrtHyujYJNtnhtnhtnOTIxZTM1MWIwZGFmYTZhYWZhNg=="
+              },
+              {
+                "eventId": "abaabcdef01234567894115b9e64da71",
+                "customerId": "exampletest",
+                "saas": "google_mail",
+                "entityId": "6fd32d1acccf146b03725eb513886822",
+                "state": "exception",
+                "type": "phishing",
+                "confidenceIndicator": "malicious",
+                "eventCreated": "2024-10-15T06:08:40.022596+00:00",
+                "severity": "5",
+                "description": "Phishing attempt detected in an email from [email protected] - 'Support: Link' ([email protected]'s mailbox)",
+                "data": "#{\"entity_id\": \"6fd32d1acccf10123456789513886822\", \"entity_type\": \"avanan_ap_scan\", \"label\": \"Phishing\"} attempt detected in an email from #{\"entity_id\": null, \"entity_type\": \"google_user\", \"disable_link\": true, \"label\": \"[email protected]\"} - '#{\"entity_id\": \"6fd32d1aabcdef012345678913886822\", \"entity_type\": \"google_mail_email\", \"label\": \"Support: Link\"}' (#{\"entity_id\": \"113301234567896535444\", \"entity_type\": \"google_user\", \"label\": \"[email protected]\"}'s mailbox)",
+                "additionalData": null,
+                "availableEventActions": null,
+                "actions": [
+                  {
+                    "actionType": "quarantine_email",
+                    "createTime": "2024-10-15T07:02:54.452012Z",
+                    "relatedEntityId": "abaabcdef01234567894115b9e64da71"
+                  },
+                  {
+                    "actionType": "google_mail_email_change_subject",
+                    "createTime": "2024-10-15T07:03:16.882267Z",
+                    "relatedEntityId": "abaabcdef01234567894115b9e64da71"
+                  },
+                  {
+                    "actionType": "restore_email",
+                    "createTime": "2024-10-15T07:03:24.885973Z",
+                    "relatedEntityId": "abaabcdef01234567894115b9e64da71"
+                  },
+                  {
+                    "actionType": "move_to_spam",
+                    "createTime": "2024-10-15T07:04:27.507214Z",
+                    "relatedEntityId": "abaabcdef01234567894115b9e64da71"
+                  },
+                  {
+                    "actionType": "google_mail_email_change_subject",
+                    "createTime": "2024-10-15T07:08:47.587776Z",
+                    "relatedEntityId": "abaabcdef01234567894115b9e64da71"
+                  }
+                ],
+                "senderAddress": "[email protected]",
+                "entityLink": "https://in.portal.checkpoint.com/dashboard/email&collaboration/CGS1?route=cHJvZmlsZS9nb29nbGregevTTevfVETgervfdvrGETGEFVefvefgqergERVeriMDM3MjVlYjUxMzg4NjgyMg=="
+              }
+            ]
+          }
+  - path: /app/hec-api/v1.0/event/query
+    methods: ['POST']
+    request_headers:
+      Content-Type:
+        - "application/json"
+      Authorization:
+        - "Bearer XnbabcdefghijklmnopqrstuvwxyztYWHWXRy7---s"
+      X-Av-Req-Id:
+        - .*
+    request_body: /^\{"requestData":\{"endDate":.*,"startDate":.*\}\}/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |-
+          {
+            "responseEnvelope": {
+              "requestId": "{{ .request.headers.Get "X-Av-Req-Id" }}",
+              "responseCode": 200,
+              "responseText": "",
+              "additionalText": "",
+              "recordsNumber": 4,
+              "scrollId": "e9c8c0e01234567890123456789000cf"
+            },
+            "responseData": [
+              {
+                "eventId": "13007abcdef01234567896d00f44a7dd4",
+                "customerId": "elasticteam",
+                "saas": "google_mail",
+                "entityId": "ad6ada66e6fabcdefabcdefc240dd9d0",
+                "state": "new",
+                "type": "phishing",
+                "confidenceIndicator": "malicious",
+                "eventCreated": "2024-10-15T09:47:06.592720+00:00",
+                "severity": "4",
+                "description": "Phishing attempt detected in an email from [email protected] - 'Office Holiday Party - Folder Share' ([email protected]'s mailbox)",
+                "data": "#{\"entity_id\": \"ad6ada66e6fabcdefabcdefc240dd9d0\", \"entity_type\": \"avanan_ap_scan\", \"label\": \"Phishing\"} attempt detected in an email from #{\"entity_id\": null, \"entity_type\": \"google_user\", \"disable_link\": true, \"label\": \"[email protected]\"} - '#{\"entity_id\": \"ad6ada66e6fabcdefabcdefc240dd9d0\", \"entity_type\": \"google_mail_email\", \"label\": \"Office Holiday Party - Folder Share\"}' (#{\"entity_id\": \"113310123456789535444\", \"entity_type\": \"google_user\", \"label\": \"[email protected]\"}'s mailbox)",
+                "additionalData": null,
+                "availableEventActions": null,
+                "actions": [],
+                "senderAddress": "[email protected]",
+                "entityLink": "https://in.portal.checkpoint.com/dashboard/email&collaboration/CGS1?route=cHJvZmlsZsdverberGRTHTYJbgbrtHYNRTGbdfbtryrtM2YyNTlkYTY0MmMyNDBkZDlkMA=="
+              },
+              {
+                "eventId": "73af5abcdef0123456789d08abfce324",
+                "customerId": "elastictest",
+                "saas": "google_mail",
+                "entityId": "4abcdefabcdefabcdefc44ca3798cb2f",
+                "state": "dismissed",
+                "type": "dlp",
+                "confidenceIndicator": "detected",
+                "eventCreated": "2024-10-15T10:27:40.294099+00:00",
+                "severity": "3",
+                "description": "SmartDLP has detected PII leak in 'Resume' ([email protected]'s mailbox)",
+                "data": "#{\"entity_id\": \"81555d1d0defabcdefabcdabcdcb6c1aa44\", \"entity_type\": \"avanan_dlp\", \"disable_link\": true, \"label\": \"SmartDLP\"} has detected #{\"entity_id\": \"81555d1d0defabcdefabcdabcdcb6c1aa44\", \"entity_type\": \"avanan_dlp\", \"disable_link\": true, \"label\": \"PII\"} leak in '#{\"entity_id\": \"40d926d7c01234567890123456789b2f\", \"entity_type\": \"google_mail_email\", \"label\": \"Resume\"}' (#{\"entity_id\": \"113310123456789535444\", \"entity_type\": \"google_user\", \"label\": \"[email protected]\"}'s mailbox)",
+                "additionalData": null,
+                "availableEventActions": null,
+                "actions": [],
+                "senderAddress": "[email protected]",
+                "entityLink": "https://in.portal.checkpoint.com/dashboard/email&collaboration/CGS1?route=cHJvZmlsZS9nb29nbGVfbWFpbF9gysdjvvsvDFGERgfvsdverERBERsdvsdv2EzNzk4Y2IyZg=="
+              }
+            ]
+          }
diff --git a/packages/checkpoint_email/changelog.yml b/packages/checkpoint_email/changelog.yml
@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+  changes:
+    - description: Initial release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11472
diff --git a/packages/checkpoint_email/data_stream/event/_dev/test/pipeline/test-common-config.yml b/packages/checkpoint_email/data_stream/event/_dev/test/pipeline/test-common-config.yml
@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_duplicate_custom_fields