Skip to content

Commit

Permalink
[7.17] [Security Solution] Fixes sorting issues related to unmapped f…
Browse files Browse the repository at this point in the history
…ields (#132190) (#132369)

## [Security Solution] Fixes sorting issues related to unmapped fields

This PR fixes the following issues related to sorting unmapped fields in timelines and the events / alerts tables:

- <#129603>
- <#123912>
- <#131625>

The `unmapped_type` property [addition](https://github.com/elastic/kibana/pull/87241/files#diff-52fd5870dcd5f783f9fc8ac3a18a8674d83ac6136e09fe0e0bcae30427d61c3fR55) to the `sort` parameter of requests was using the `type` field metadata from `BrowserFields`, but the `type` metadata (for some fields) contains the value `string`, which is not a [valid field data type](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html).

The fix for the issues above:

- Populates the `sort` property of requests with values from the `esTypes` `BrowserFields` metadata (instead of `type`)
  - The `esTypes` metadata may specify more than one field value type. When `esTypes` contains more than one type, and `keyword` is one of the types, the `sort` property of the request will prefer `keyword` over other the other types
- When the field metadata has an empty `esTypes` collection, the `sort` property of the request will default to using `"unmapped_type": "keyword"`
- The field type displayed in tooltips when hovering over columns in a timeline now displays values from `esTypes` instead of `type`

### Desk testing

To reproduce issue <#129603> and to verify the fix:

1) Open Kibana `Dev tools`

2) Execute the following query to delete any exiting `logs-ti_test` index:

```
DELETE logs-ti_test
```

3) Execute the following query to create an index named `logs-ti_test`, which has the following properities:

- Dynamic mappings are disabled via `"dynamic": false`
- It does NOT contain a mapping for `event.action` (we will sort by this field in later steps)
- It contains a mapping for the non-ECS `testing` field

```
PUT logs-ti_test
{
  "mappings": {
    "dynamic": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "event": {
        "properties": {
          "category": {
            "type": "keyword"
          },
          "dataset": {
            "type": "keyword"
          },
          "kind": {
            "type": "keyword"
          },
          "type": {
            "type": "keyword"
          }
        }
      },
      "host": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "testing": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "threat": {
        "properties": {
          "indicator": {
            "properties": {
              "file": {
                "properties": {
                  "hash": {
                    "properties": {
                      "md5": {
                        "type": "keyword"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

4) Execute the following query to add a new document to the `logs-ti_test` index, and note that:

- It does NOT contain a `event.action` field
- It contains a value for the non-ECS `testing` field

```
POST logs-ti_test/_doc/
{
  "@timestamp": "2022-05-12T00:00:14.725Z",
  "host": {
    "name": "foozle"
  },
  "threat": {
    "indicator": {
      "file": {
        "hash": {
          "md5": "a4f87cbcd2a4241da77b6bf0c5d9e8553fec991f"
        }
      }
    }
  },
  "event": {
    "kind": "enrichment",
    "type": "indicator",
    "dataset": "ti_*",
    "category": "threat"
  },
  "testing": "simulated threat intel data"
}
```

5) Navigate to the Security > Hosts page

6) Select `Last 1 year` from the date picker

7) Click the `Events` tab

8) Enter the following KQL query in the search bar at the top of the page:

```
host.name: foozle
```

9) Hover over the `foozle` entry in the `host.name` column in the Events table, and click the `Add to timeline investigation` cell action

10) Open the timeline

11) Hover over the `event.action` field

**Expected result**

- The tooltip displays  type `keyword` for the `event.action` field

**Actual result**

- The tooltip displays type `string` for the `event.action` field

12) Click the `event.action` column to add a secondary sort

**Expected result**

- The table is sorted by `@timestamp` and `event.action`
- The table contents are (still) visible

**Actual result**

- The table is sorted by `@timestamp` and `event.action`
- The contents of the table are now empty

13) Click the timeline's `Inspect` button

14) In the `Inspect Timeline` dialog, click the `Request` tab

15) Scroll down to the `sort` property of the request

**Expected result**

- The `event.action` field contains a `"unmapped_type": "keyword"` property, per the example below:

```json
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "date"
      }
    },
    {
      "event.action": {
        "order": "desc",
        "unmapped_type": "keyword"
      }
    }
  ],
  ```

**Actual result**

- The request's `event.action` field contains a `"unmapped_type": "string"` property, per the example below:

```json
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "number"
      }
    },
    {
      "event.action": {
        "order": "desc",
        "unmapped_type": "string"
      }
    }
  ],
  ```

16) In the `Inspect Timeline` dialog, click the `Response` tab

**Expected result**

- The response contains `0` `failed` shards / no failures

**Actual result**

- The response contains failures for the `logs-ti_test` index, with the following reason:

```
"reason": "No mapper found for type [string]"
```

per the example below:

```json
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 4,
    "successful": 3,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "logs-ti_test",
        "node": "NCRcGeDqSlKQiuPWVFvMEg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "No mapper found for type [string]"
        }
      }
    ]
  },
```
  • Loading branch information
andrew-goldstein committed May 18, 2022
1 parent bfa1d73 commit a408358
Show file tree
Hide file tree
Showing 70 changed files with 1,027 additions and 180 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ export interface BrowserField {
name: string;
searchable: boolean;
type: string;
esTypes?: string[];
subType?: IFieldSubType;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ export interface TimelineRequestBasicOptions extends IEsSearchRequest {

export interface TimelineRequestSortField<Field = string> extends SortField<Field> {
type: string;
esTypes: string[];
}

export interface TimelineRequestOptionsPaginated<Field = string>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ export type SortDirection = 'none' | 'asc' | 'desc' | Direction;
export interface SortColumnTimeline {
columnId: string;
columnType: string;
esTypes?: string[];
sortDirection: SortDirection;
}

Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ export const defaultHeaders: ColumnHeaderOptions[] = [
columnHeaderType: defaultColumnHeaderType,
id: '@timestamp',
initialWidth: DEFAULT_DATE_COLUMN_MIN_WIDTH,
esTypes: ['date'],
type: 'date',
},
{
columnHeaderType: defaultColumnHeaderType,
Expand Down
Loading

0 comments on commit a408358

Please sign in to comment.