feat: add support for multiple key grouping in count condition

[fukusuket]

What Changed

• Closed Support multiple grouping by in count #1339

Evidence

multiple key grouping

I run with the following rules,
./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -A -a -D -n -u -q -r test.yml

title: TEST_TITLE
details: ''  #Cannot be used because this is a count rule
description: TEST

id: 5e00cf44-84bb-4083-8ae6-213005b2307a
level: informational
status: test
detection:
    selection:
        Channel: Security
        EventID: 4625
    filter:
       IpAddress: "-"
    condition: selection and not filter | count() by IpAddress,SubStatus,LogonType >= 2
    timeframe: 5d

then I confirmed multiple key grouping works as follows.

Timestamp · RuleTitle · Level · Computer · Channel · EventID · RecordID · Details · ExtraFieldInfo
2016-09-20 01:50:06.513 +09:00 · TEST_TITLE · info · - · - · - ·  · Count: 3558 ¦ IpAddress: 192.168.198.149 ¦ SubStatus: 0xc000006a ¦ LogonType: 3 ·
2016-09-20 01:50:08.000 +09:00 · TEST_TITLE · info · - · - · - ·  · Count: 2 ¦ IpAddress: 192.168.198.149 ¦ SubStatus: 0xc0000072 ¦ LogonType: 3 ·
2021-10-24 06:50:11.666 +09:00 · TEST_TITLE · info · - · - · - ·  · Count: 2 ¦ IpAddress: 10.23.23.9 ¦ SubStatus: 0x0 ¦ LogonType: 10 ·

I would appreciate it if you could check it out when you have time🙏

[fukusuket]

Check #1341 (comment) result

I confirmed the validity of #1341 (comment) result with the following results.

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -A -a -r check.yml -D -n -u -q -o timeline.csv -C

title: Checking
details: ''  #Cannot be used because this is a count rule
description: Checking

id: 45e8a0fc-60c2-46d7-ba39-aafb15b9854e
level: medium
status: stable
logsource:
    product: windows
    service: security
detection:
    selection:
        Channel: Security
        EventID: 4625
    filter:
       IpAddress: "-"
    condition: selection and not filter

% cat timeline.csv | wc -l
    3564
% cat timeline.csv | grep 192.168.198.149 | grep "WRONG PW" | wc -l  # count rule does not support field value conversion
    3558
% cat timeline.csv | grep 192.168.198.149 | grep 0xc0000072 | wc -l
       2
% cat timeline.csv | grep 10.23.23.9 | wc -l
       2

[fukusuket]

#1339 (comment)

I confirmed that the output has changed as shown below.

before:

% ./hayabusa-2.15.0-mac-aarch64 csv-timeline -d ../hayabusa-sample-evtx -D -n -u -w -r rules/sigma/builtin/unsupported/win_security_susp_failed_logons_single_source_kerberos3.yml -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RecordID · Details · ExtraFieldInfo
2021-12-02 23:48:15.983 +09:00 · Invalid Users Failing To Authenticate From Source Using Kerberos · med · - · - · - ·  · [condition] count(TargetUserName) by IpAddress > 10 in timeframe [result] count: 46 TargetUserName:syvsdy/vase/xvtrz/admtest/gsdf/g/tfay/rec/xt/ysy/vs/rey/tbyt/go/b aer/mgdi/xc/sef/yvsyv/sfs/test2/accrt/vga/srey/nd/ryver/wyt/s/vrat/m,og/vt/bdcy/tc/nini/ytuntsr/sgfg/tary/test1/vdr/dyfgdhbn/bsfin/yvas/ar/uydzry/ugu/vay IpAddress:::ffff:10.23.123.11 timeframe:24h ·

after:

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -D -n -u -w -r rules/sigma/builtin/unsupported/win_security_susp_failed_logons_single_source_kerberos3.yml -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RecordID · Details · ExtraFieldInfo
2021-12-02 23:48:15.983 +09:00 · Invalid Users Failing To Authenticate From Source Using Kerberos · med · - · - · - ·  · Count: 46 ¦ TargetUserName: syvsdy/vase/xvtrz/admtest/gsdf/g/tfay/rec/xt/ysy/vs/rey/tbyt/go/b aer/mgdi/xc/sef/yvsyv/sfs/test2/accrt/vga/srey/nd/ryver/wyt/s/vrat/m,og/vt/bdcy/tc/nini/ytuntsr/sgfg/tary/test1/vdr/dyfgdhbn/bsfin/yvas/ar/uydzry/ugu/vay ¦ IpAddress: ::ffff:10.23.123.11 ·

[YamatoSecurity]

@fukusuket Thanks so much! Very nice! Just a couple additional things.

I am testing with this: ./target/release/hayabusa csv-timeline -d ../hayabusa-sample-evtx -w -r ~/Desktop/test.yml -p verbose

and get these results:

Timestamp · RuleTitle · Level · Computer · Channel · EventID · MitreTactics · MitreTags · OtherTags · RecordID · Details · ExtraFieldInfo · RuleFile · EvtxFile
2016-09-20 01:50:06.513 +09:00 · TEST_TITLE · info · - · - · - ·  ·  ·  ·  · Count: 3558 ¦ IpAddress: 192.168.198.149 ¦ SubStatus: 0xc000006a ¦ LogonType: 3 ·· test.yml · -

2016-09-20 01:50:08.000 +09:00 · TEST_TITLE · info · - · - · - ·  ·  ·  ·  · Count: 2 ¦ IpAddress: 192.168.198.149 ¦ SubStatus: 0xc0000072 ¦ LogonType: 3 ·· test.yml · -

2021-10-24 06:50:11.666 +09:00 · TEST_TITLE · info · - · - · - ·  ·  ·  ·  · Count: 2 ¦ IpAddress: 10.23.23.9 ¦ SubStatus: 0x0 ¦ LogonType: 10 ·· test.yml · -

1. Is it possible to put in the Channel and EventID info? When there are multiple values, we can separate them with ¦.
2. ExtraFieldInfo is blank (··) but would nice to have it as - which we use for n/a. So ·· -> · - ·
3. EvtxFile is also - but many users ask me how they can look up the evtx file from this alert so it would be nice to get this information as well. Again, separate by ¦ when there are multiple values.
4. Is it possible to do the field key mapping? For example, change SubStatus: 0xc000006a to SubStatus: WRONG PW, etc...

If any of these are difficult we can separate to a different issue.

Also, I found one bigger issue. The results are not outputted with json-timeline:

{
    "Timestamp": "2021-10-24 06:50:11.666 +09:00",
    "RuleTitle": "TEST_TITLE",
    "Level": "info",
    "Computer": "-",
    "Channel": "-",
    "EventID": "-",
    "RecordID": "",
    "Details": "",
    "ExtraFieldInfo": "-",
    "RuleFile": "test.yml",
    "EvtxFile": "-"
}

[fukusuket]

@YamatoSecurity
Thank you so much for testing :)
Since 1-4 above are changes that are different from the multiple key grouping, I think it might be better to deal with them as separate issues, so I will create another issue!

Also, I found one bigger issue. The results are not outputted with json-timeline:

This is a bug... :( so we'll fix it!

[fukusuket]

@YamatoSecurity

Also, I found one bigger issue. The results are not outputted with json-timeline:

I fixed above case as follows. Could you confirm it?🙏

% ./hayabusa csv-timeline -d ../hayabusa-sample-evtx -D -n -u -w -r test2.yml -q
...
Timestamp · RuleTitle · Level · Computer · Channel · EventID · RecordID · Details · ExtraFieldInfo
2016-09-20 01:50:06.513 +09:00 · TEST_TITLE · info · - · - · - ·  · Count: 3558 ¦ IpAddress: 192.168.198.149 ¦ SubStatus: 0xc000006a ¦ LogonType: 3 ·
2016-09-20 01:50:08.000 +09:00 · TEST_TITLE · info · - · - · - ·  · Count: 2 ¦ IpAddress: 192.168.198.149 ¦ SubStatus: 0xc0000072 ¦ LogonType: 3 ·
2021-10-24 06:50:11.666 +09:00 · TEST_TITLE · info · - · - · - ·  · Count: 2 ¦ IpAddress: 10.23.23.9 ¦ SubStatus: 0x0 ¦ LogonType: 10 ·

% ./hayabusa json-timeline -d ../hayabusa-sample-evtx -D -n -u -w -r test2.yml -q
...
{
    "Timestamp": "2016-09-20 01:50:06.513 +09:00",
    "RuleTitle": "TEST_TITLE",
    "Level": "info",
    "Computer": "-",
    "Channel": "-",
    "EventID": "-",
    "RecordID": "",
    "Details": "Count:3558 ¦ IpAddress:192.168.198.149 ¦ SubStatus:0xc000006a ¦ LogonType:3",
    "ExtraFieldInfo": "-"
}
{
    "Timestamp": "2016-09-20 01:50:08.000 +09:00",
    "RuleTitle": "TEST_TITLE",
    "Level": "info",
    "Computer": "-",
    "Channel": "-",
    "EventID": "-",
    "RecordID": "",
    "Details": "Count:2 ¦ IpAddress:192.168.198.149 ¦ SubStatus:0xc0000072 ¦ LogonType:3",
    "ExtraFieldInfo": "-"
}
{
    "Timestamp": "2021-10-24 06:50:11.666 +09:00",
    "RuleTitle": "TEST_TITLE",
    "Level": "info",
    "Computer": "-",
    "Channel": "-",
    "EventID": "-",
    "RecordID": "",
    "Details": "Count:2 ¦ IpAddress:10.23.23.9 ¦ SubStatus:0x0 ¦ LogonType:10",
    "ExtraFieldInfo": "-"
}

[YamatoSecurity]

@fukusuket Thanks so much! How difficult is it to output in the following way?:

"Details": {
    "Count": 2,
    "IpAddress": "10.23.23.9",
    "SubStatus": "0x0",
    "LogonType": "10"
}

If it requires many changes then we can leave it as it is for now.

[fukusuket]

@YamatoSecurity
Thank you for checking! I think it's probably possible, but I think it might be better to create it while imagining the specifications to some extent in #1342, so I'd appreciate it if you could leave it as is!

[fukusuket]

For example, we should consider the following:

• If top level Channel has multiple values, should it be an array or string or -?
• If top level EventID has multiple values, should it be an array or string or -?
• If each field under Details has multiple values, should it be an array or string?
• Should the aggregate results of Channel and EventID be placed under Details?

(memo: Takajo expects the top level field to be a single value)

[YamatoSecurity]

For example, we should consider the following:

• If top level Channel has multiple values, should it be an array or string or -?
• If top level EventID has multiple values, should it be an array or string or -?
• If each field under Details has multiple values, should it be an array or string?
• Should the aggregate results of Channel and EventID be placed under Details?

These are all very good questions! Indeed, let's think about this and how it affects JSON importing, etc..

[YamatoSecurity]

@fukusuket LGTM!