Bot development: intelmq.lib.exceptions.KeyExists

[vandaref]

I'm trying to create a custom parser bot for Crowdstrike data collected.

I'm facing the following issue :
intelmq.lib.exceptions.KeyExists: key 'malware.hash.sha256' already exists
and it could happens with others fields than malware.hash.sha256.

I suppose there are duplicate hash (or other fields) in the feed I'm collecting.
This is the output of my debug :

rowdstrike-parser: Start loop
crowdstrike-parser: 91c03c1acf3d1dfee8aa458d08cc51b5ec7c8708
crowdstrike-parser: End loop
crowdstrike-parser: Start loop
crowdstrike-parser: End loop
crowdstrike-parser: Start loop
crowdstrike-parser: 7c3f83d9ebc4adcc2e76813de65450deffc225633806a071036cb36bb3afaa60
crowdstrike-parser: End loop
crowdstrike-parser: Start loop
crowdstrike-parser: c840db85a0f4a469f471d494de86828f24a1e841fed3719b52e1f6d13e5f4616
crowdstrike-parser: Failed to parse line. 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/intelmq/lib/bot.py", line 1225, in process
    events: list[libmessage.Event] = list(filter(bool, value))
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/intelmq/bots/parsers/crowdstrike/parser.py", line 36, in parse_line
    event.add("malware.hash.sha256", ioc["indicator"], raise_failure=False)
  File "/usr/lib/python3/dist-packages/intelmq/lib/message.py", line 237, in add
    raise exceptions.KeyExists(key)
intelmq.lib.exceptions.KeyExists: key 'malware.hash.sha256' already exists

[sebix]

You cannot set a key if they key already exists without explicitly overwriting it. Use event.add(key, value, overwrite=True) instead.
https://intelmq.readthedocs.io/en/develop/source/intelmq.lib.html#intelmq.lib.message.Message.add

[vandaref]

Will it erase each new value and at the end I'll have only the last one ?

[sebix]

Yes, each key can only have one value. It's a dictionary. https://docs.intelmq.org/latest/user/event/#fields-reference lists all fields and their type.

If you need to write lists, you can use a custom field in the extra. namespace. They can have any type.

[vandaref]

But each event is different. I don't have this issue with other bot. The hash of each document will be different. So the key already exists with an other value.
Please see this example :

{
  "id": "hash_md5_58a5bdcf325429d36194202544359f22",
  "indicator": "58a5bdcf325429d36194202544359f22",
  "type": "hash_md5",
  "deleted": false,
  "published_date": 1364395570,
  "last_updated": 1707145213,
  ...,
  ],
  "vulnerabilities": []
},
{
  "id": "hash_md5_ad7eacf53192afdce79b951ba860d3d3",
  "indicator": "ad7eacf53192afdce79b951ba860d3d3",
  "type": "hash_md5",
  "deleted": false,
  "published_date": 1378907777,
  "last_updated": 1707145213,
  ..,
  "vulnerabilities": []
}

I implement indicator value as malware.hash.XXX key. I'm not sure to understand because on other bot this is the same schema we have different value for one key.

[sebix]

But each event is different. I don't have this issue with other bot. The hash of each document will be different.

And each "document" will become its own event? If every document has its own hash, there will be no conflicts.

I implement indicator value as malware.hash.XXX key. I'm not sure to understand because on other bot this is the same schema we have different value for one key.

We don't know your bot code.

[vandaref]

Please have a look on my parser bot here :

# SPDX-FileCopyrightText: 2020 sinus-x
#
# SPDX-License-Identifier: AGPL-3.0-or-later

# -*- coding: utf-8 -*-
import json

from intelmq.lib.harmonization import DateTime
from intelmq.lib.bot import ParserBot
from intelmq.lib.utils import base64_decode


class CrowdstrikeParserBot(ParserBot):

    recover_line = ParserBot.recover_line_json

    def parse(self, report):

        return [json.loads(base64_decode(report["raw"]))]

    def parse_line(self, raw_json, report):
        event = self.new_event(report)

        # self.logger.debug(str(type(raw_json)))
        # self.logger.debug(json.dumps(raw_json))

        if "resources" in raw_json:
            for ioc in raw_json["resources"]:
                self.logger.debug(str("Start loop"))
                if "type" in ioc and ioc["type"] == "hash_sha1":
                    self.logger.debug(str(ioc["indicator"]))
                    try:
                        event.add("malware.hash.sha1", ioc["indicator"], raise_failure=False)
                    except:
                        self.logger.warning(f"Exception, sha1 not added.")
                        pass

                elif "type" in ioc and ioc["type"] == "hash_sha256":
                    self.logger.debug(str(ioc["indicator"]))
                    try:
                        event.add("malware.hash.sha256", ioc["indicator"], raise_failure=False)
                    except:
                        self.logger.warning(f"Exception, sha256 not added.")
                        pass

                elif "type" in ioc and ioc["type"] == "hash.md5":
                    self.logger.debug(str(ioc["indicator"]))
                    try:
                        event.add("malware.hash.md5", ioc["indicator"], raise_failure=False)
                    except:
                        self.logger.warning(f"Exception, md5 not added.")
                        pass

                if "published_date" in ioc and ioc["published_date"]:
                    self.logger.debug(str(DateTime.from_timestamp(ioc["published_date"])))
                    try:
                        event.update("time.source", str(DateTime.from_timestamp(ioc["published_date"])))
                    except:
                        self.logger.warning(f"Exception, time.source not added.")
                        pass

                self.logger.debug(str("End loop"))
            yield event

BOT = CrowdstrikeParserBot

And the message incomming is this from the collector bot :
{
"feed.accuracy": 100.0,
"feed.name": "Crowdstrike Collector",
"feed.provider": "crowdstrike.com",
"feed.url": "https://api.eu-1.crowdstrike.com/intel/combined/indicators/v1?limit=10000&sort=published_date%7Casc&filter=last_updated%3A%3E%3D1707144665&include_deleted=false&include_relations=false",
"raw": "ewogIm1......",
"time.observation": "2024-02-05T14:51:44+00:00"
}

I'm going through the raw line and it looks like I sent just before. As i wrote the code yes each document should be its own document. So this is why I don't understand the key issue.

Please feel free to ask me more details or precision.

[sebix]

I don't have the input data, so it's again a guess.
raw_json["resources"] appears to be a list not a dict of lists. Thus, in each loop iteration event already has some data.
Try moving the event = and yield event inside the loop.

[vandaref]

Please have a look on my output when I run the command intelmqctl run crowdstrike-parser -l DEBUG

crowdstrike-parser: Start loop
crowdstrike-parser: 1aa02fe47d566823b36b3e438e52cb8c2676d267bc11da4020befb3035d56b84
crowdstrike-parser: Exception, sha256 not added.
crowdstrike-parser: 2024-02-10T22:46:09+00:00
crowdstrike-parser: Exception, time.source not added.
crowdstrike-parser: End loop
crowdstrike-parser: Start loop
crowdstrike-parser: 2024-02-10T22:46:09+00:00
crowdstrike-parser: Exception, time.source not added.
crowdstrike-parser: End loop
crowdstrike-parser: Start loop
crowdstrike-parser: 79fdf75bead3fe60c934cb3d2495d12a4d41ebdb
crowdstrike-parser: Exception, sha1 not added.
crowdstrike-parser: 2024-02-10T23:15:54+00:00
crowdstrike-parser: Exception, time.source not added.
crowdstrike-parser: End loop

If I move the event = and yield event inside the loop the output message looks way better :

{
    "feed.accuracy": 100.0,
    "feed.name": "Crowdstrike Collector",
    "feed.provider": "crowdstrike.com",
    "feed.url": "https://api.eu-1.crowdstrike.com/intel/combined/indicators/v1?limit=10000&sort=published_date%7Casc&filter=last_updated%3A%3E%3D1707726284&include_deleted=false&include_relations=false",
    "malware.hash.sha256": "a8b9befd8715d641ee2ad3eb5f10cec1031ec358b2cc7291feb29a5a6cd3b377",
    "time.observation": "2024-02-12T09:24:44+00:00"
} 

[vandaref]

I used this bot https://github.com/certtools/intelmq/blob/develop/intelmq/bots/parsers/cznic/parser_proki.py as a "template". Maybe you can have more context with this. I should send it before...